Jul 13 23:37:15.656781 ip-10-0-128-174 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 23:37:15.656791 ip-10-0-128-174 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 23:37:15.656799 ip-10-0-128-174 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 23:37:15.657013 ip-10-0-128-174 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 23:37:24.614346 ip-10-0-128-174 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 23:37:24.614360 ip-10-0-128-174 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e6613ced032c4ba09b84ae65b903dee4 --
Jul 13 23:39:11.595141 ip-10-0-128-174 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.000937    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004305    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004316    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004320    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004323    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004326    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004330    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004332    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.088072 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004335    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004338    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004340    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004343    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004345    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004348    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004351    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004354    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004357    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004359    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004362    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004365    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004375    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004378    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004380    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004383    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004386    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004388    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004391    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004393    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.088816 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004396    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004398    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004401    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004403    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004406    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004409    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004411    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004414    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004416    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004419    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004422    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004424    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004427    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004430    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004432    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004435    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004437    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004440    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004443    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.089396 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004445    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004448    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004450    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004453    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004456    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004458    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004462    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004464    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004466    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004469    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004471    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004475    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004479    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004482    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004485    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004487    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004490    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004493    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004496    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004498    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.089908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004501    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004503    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004506    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004509    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004512    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004514    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004518    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004520    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004523    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004526    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004529    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004531    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004534    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004538    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004540    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004544    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004547    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004549    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004552    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004555    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.090463 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004558    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004560    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004563    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004566    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004569    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004573    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004575    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004579    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004583    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004586    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004589    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004592    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.004595    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006238    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006246    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006249    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006252    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006256    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006259    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.090998 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006262    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006265    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006268    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006272    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006274    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006277    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006280    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006283    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006286    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006288    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006292    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006296    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006300    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006303    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006305    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006308    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006311    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006313    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006316    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.091547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006321    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006324    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006327    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006330    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006333    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006336    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006339    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006342    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006344    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006347    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006350    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006354    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006357    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006360    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006363    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006366    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006369    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006372    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006374    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006377    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.092109 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006379    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006382    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006384    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006387    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006390    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006393    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006395    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006399    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006402    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006404    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006406    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006410    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006412    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006415    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006417    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006420    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006422    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006424    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006427    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006429    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.092667 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006432    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006434    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006438    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006440    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006443    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006445    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006450    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006453    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006456    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006459    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006461    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006465    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006467    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006470    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006472    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006474    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006477    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006480    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006482    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.093207 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006485    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006487    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006490    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006493    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006495    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006498    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006500    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006503    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006505    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006507    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006510    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006513    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006515    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006518    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006520    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006858    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006862    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006865    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006868    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006871    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.093970 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006874    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.094314 ip-10-0-128-174 systemd[1]: Started Kubernetes Kubelet.
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006876    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006879    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006897    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006901    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006903    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006906    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006908    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006914    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006917    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006920    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006922    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006925    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006928    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006931    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006933    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006936    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006938    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006941    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006943    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.094911 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006946    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006948    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006951    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006953    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006956    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006959    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006961    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006963    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006966    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006968    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006971    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006973    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006976    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006978    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006981    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006983    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006986    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006988    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006991    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006993    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.096244 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.006998    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007001    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007003    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007006    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007008    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007011    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007014    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007018    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007022    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007024    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007027    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007029    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007032    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007035    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007037    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007039    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007042    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007045    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007047    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.097469 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007050    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007052    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007056    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007060    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007063    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007066    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007069    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007072    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007074    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007077    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007080    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007082    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007085    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007092    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007094    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007097    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007101    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007104    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007107    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.098344 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007110    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007113    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007116    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007118    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007121    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007124    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007127    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007129    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007132    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007135    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007138    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007141    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007143    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007146    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007148    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007151    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007217    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007224    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007230    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007235    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007239    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 23:39:12.099169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007242    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007247    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007250    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007254    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007257    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007267    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007270    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007273    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007276    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007279    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007281    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007284    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007287    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007291    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007294    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007297    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007300    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007304    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007307    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007310    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007313    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007317    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007319    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007322    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 23:39:12.100234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007325    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007328    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007331    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007334    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007337    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007340    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007343    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007346    2558 flags.go:64] FLAG: --enable-server="true"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007349    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007354    2558 flags.go:64] FLAG: --event-burst="100"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007357    2558 flags.go:64] FLAG: --event-qps="50"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007360    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007363    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007369    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007373    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007376    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007379    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007382    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007385    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007387    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007391    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007394    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007397    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007400    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007403    2558 flags.go:64] FLAG: --feature-gates=""
Jul 13 23:39:12.101070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007407    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007409    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007412    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007416    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007418    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007421    2558 flags.go:64] FLAG: --help="false"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007424    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-128-174.ec2.internal"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007427    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007430    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007433    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007436    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007439    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007442    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007445    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007447    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007450    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007453    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007456    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007459    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007462    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007468    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007471    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007474    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007477    2558 flags.go:64] FLAG: --lock-file=""
Jul 13 23:39:12.102239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007479    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007483    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007486    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007490    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007493    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007496    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007499    2558 flags.go:64] FLAG: --logging-format="text"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007502    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007506    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007509    2558 flags.go:64] FLAG: --manifest-url=""
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007511    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007515    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007518    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007522    2558 flags.go:64] FLAG: --max-pods="110"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007525    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007528    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007530    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007533    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007536    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007540    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007542    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007549    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007552    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007555    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 23:39:12.194029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007558    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007561    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007566    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007569    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007575    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007579    2558 flags.go:64] FLAG: --port="10250"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007582    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007584    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0830c6f41e10082ae"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007587    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007590    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007593    2558 flags.go:64] FLAG: --register-node="true"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007596    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007600    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007602    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007605    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007608    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007612    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007615    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007618    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007631    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007635    2558 flags.go:64] FLAG: --runonce="false"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007638    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007641    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007644    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007647    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 23:39:12.195508 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007650    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007653    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007656    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007659    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007661    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007664    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007667    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007670    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007673    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007675    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007678    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007685    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007689    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007692    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007696    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007698    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007701    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007704    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007707    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007710    2558 flags.go:64] FLAG: --v="2"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007715    2558 flags.go:64] FLAG: --version="false"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007718    2558 flags.go:64] FLAG: --vmodule=""
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007722    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.007726    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007836    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.196727 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007839    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007843    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007846    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007849    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007851    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007855    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007859    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007862    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007865    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007868    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007870    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007873    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007875    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007878    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007880    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007901    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007904    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007907    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007911    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.198139 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007915    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007918    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007920    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007923    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007926    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007928    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007931    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007934    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007937    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007939    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007942    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007945    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007948    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007950    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007953    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007956    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007960    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007963    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007965    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007968    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.198855 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007970    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007973    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007976    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007978    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007981    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007983    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007986    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007988    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007991    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007993    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007996    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.007999    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008004    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008007    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008009    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008012    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008014    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008017    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008021    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008025    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.199992 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008028    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008031    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008033    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008036    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008039    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008044    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008047    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008050    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008052    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008054    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008057    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008060    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008062    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008065    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008067    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008070    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008072    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008075    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008077    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008080    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.200736 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008082    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008085    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008087    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008091    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008094    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008097    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008099    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008102    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008104    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008107    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008109    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008112    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008114    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008117    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008119    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008122    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008124    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008127    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.206721 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.008130    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.008739    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.019760    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.019775    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.020572    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.020579    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020635    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020640    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020644    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020647    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020650    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020654    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020657    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020659    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020662    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.207523 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020665    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020667    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020670    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020673    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020678    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020681    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020684    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020687    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020689    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020692    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020695    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020698    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020700    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020703    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020705    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020708    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020710    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020713    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020716    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.208995 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020718    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020721    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020723    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020727    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020729    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020732    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020734    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020737    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020740    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020743    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020745    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020749    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020753    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020756    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020759    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020763    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020765    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020770    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020773    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020776    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.209741 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020779    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020782    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020784    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020787    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020789    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020792    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020795    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020797    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020800    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020803    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020822    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020825    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020828    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020831    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020834    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020838    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020840    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020843    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020845    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020848    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.210850 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020851    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020853    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020856    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020858    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020861    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020864    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020866    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020868    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020871    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020875    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020878    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020881    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020895    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020898    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020901    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020903    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020906    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020909    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020913    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020916    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.307753 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020919    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020922    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020925    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020928    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020930    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020933    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020935    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020939    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020942    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020944    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.020948    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.020954    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021060    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021066    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 23:39:12.308952 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021069    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021072    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021076    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021080    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021084    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021087    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021089    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021092    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021096    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021099    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021102    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021104    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021107    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021110    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021113    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021116    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021118    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021121    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021124    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 23:39:12.309751 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021127    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021130    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021132    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021135    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021137    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021140    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021144    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021146    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021149    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021151    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021154    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021157    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021159    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021162    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021164    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021167    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021169    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021172    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021174    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021177    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 23:39:12.311781 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021179    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021183    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021185    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021188    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021191    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021194    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021196    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021199    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021202    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021204    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021207    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021209    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021212    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021214    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021217    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021219    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021223    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021226    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021230    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 23:39:12.312585 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021233    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021236    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021239    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021241    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021244    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021246    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021249    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021251    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021254    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021257    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021259    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021261    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021264    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021266    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021270    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021273    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021275    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021278    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021281    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021283    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 23:39:12.313601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021286    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021288    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021291    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021293    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021296    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021298    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021301    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021304    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021306    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021309    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021311    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021315    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021318    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021320    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021323    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021325    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021329    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021332    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 23:39:12.316435 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:12.021335    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.021340    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.021434    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.023385    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.024583    2558 server.go:1019] "Starting client certificate rotation"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.024687    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.024725    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.027016    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.030996    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.046125    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.051242    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.052778    2558 log.go:25] "Validated CRI v1 image API"
Jul 13 23:39:12.317239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.053588    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 23:39:12.318855 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.058742    2558 fs.go:135] Filesystem UUIDs: map[293cce98-da8a-454f-829f-0218e8e4ee41:/dev/nvme0n1p4 3f9b1cbe-f5b9-4a33-ae62-bc684c108c70:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 13 23:39:12.318855 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.058756    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.062713    2558 manager.go:216] Machine: {Timestamp:2026-07-13 23:39:12.062248257 +0000 UTC m=+0.349466594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099705 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cc941e19369da2f27538f7a44e086 SystemUUID:ec2cc941-e193-69da-2f27-538f7a44e086 BootID:e6613ced-032c-4ba0-9b84-ae65b903dee4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:97:0c:dc:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:97:0c:dc:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:e9:f9:aa:6a:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.062810    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.062880    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.068947    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtdp9"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.069354    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.069379    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-174.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.070381    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.070401    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.070414    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.070998    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.072564    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075380    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075493    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075520    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075529    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075542    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 23:39:12.320892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075554    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075565    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.075652    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vtdp9"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.076608    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.076642    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.079041    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.085250    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088181    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088261    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088291    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088445    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088458    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088464    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088469    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088477    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088482    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088869    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088875    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088881    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088900    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088908    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.088915    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.090549    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.090557    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.093508    2558 server.go:1268] "Started kubelet"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.093584    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.093580    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.094039    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.095520    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.095522    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.096373    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.099185    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.099653    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:12.099754    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-174.ec2.internal\" not found"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.100258    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.100275    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.100372    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.100381    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.100642    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101034    2558 factory.go:55] Registering systemd factory
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101054    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101299    2558 factory.go:153] Registering CRI-O factory
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101313    2558 factory.go:223] Registration of the crio container factory successfully
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101391    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101418    2558 factory.go:103] Registering Raw factory
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101435    2558 manager.go:1196] Started watching for new ooms in manager
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101533    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.101849    2558 manager.go:349] Starting recovery of all containers
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:12.109245    2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-174.ec2.internal\" not found" node="ip-10-0-128-174.ec2.internal"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.118144    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:12.120094    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 23:39:12.321728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:12.175333    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:12.323398 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:12.200549    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-174.ec2.internal\" not found"
Jul 13 23:39:13.298133 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.298103    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-174.ec2.internal\" not found"
Jul 13 23:39:13.298622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.298192    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 23:39:13.298622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.298240    2558 apiserver.go:52] "Watching apiserver"
Jul 13 23:39:13.298622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.298304    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 23:34:12 +0000 UTC" deadline="2028-04-03 10:50:35.3102349 +0000 UTC"
Jul 13 23:39:13.298622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.298356    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15107h11m22.011883058s"
Jul 13 23:39:13.302368 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.302347    2558 manager.go:354] Recovery completed
Jul 13 23:39:13.307299 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.307283    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 23:39:13.309488 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.309460    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 23:39:13.309578 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.309496    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 23:39:13.309578 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.309513    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientPID"
Jul 13 23:39:13.310162 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310149    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 23:39:13.310162 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310160    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 23:39:13.310298 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310169    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 23:39:13.310298 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310189    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 23:39:13.310357 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310303    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:13.310357 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.310349    2558 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:13.310605 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310590    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 23:39:13.311018 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.310999    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 23:39:13.311109 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.311030    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 23:39:13.311109 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.311045    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientPID"
Jul 13 23:39:13.311575 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.311561    2558 policy_none.go:49] "None policy: Start"
Jul 13 23:39:13.311610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.311580    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 23:39:13.311610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.311592    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 23:39:13.312747 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.312735    2558 policy_none.go:47] "Start"
Jul 13 23:39:13.343693 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.343575    2558 manager.go:337] "Starting Device Plugin manager"
Jul 13 23:39:13.343775 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.343722    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 23:39:13.343775 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.343737    2558 server.go:87] "Starting device plugin registration server"
Jul 13 23:39:13.344133 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.344118    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 23:39:13.344199 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.344134    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 23:39:13.344267 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.344248    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 23:39:13.344368 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.344349    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 23:39:13.344368 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.344364    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 23:39:13.344929 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.344904    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 23:39:13.345039 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.344952    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-174.ec2.internal\" not found"
Jul 13 23:39:13.348735 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.348721    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:13.366302 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.366286    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:13.424359 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.424307    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 23:39:13.425545 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.425521    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 23:39:13.425623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.425557    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 23:39:13.425623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.425580    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 23:39:13.425707 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.425622    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 23:39:13.426104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.426083    2558 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-128-174.ec2.internal" not found
Jul 13 23:39:13.427879 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.427865    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 23:39:13.444916 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.444901    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 23:39:13.447114 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.447099    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 23:39:13.447186 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.447129    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 23:39:13.447186 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.447142    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeHasSufficientPID"
Jul 13 23:39:13.447186 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.447177    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.456598 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.456582    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.456669 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.456603    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-174.ec2.internal\": node \"ip-10-0-128-174.ec2.internal\" not found"
Jul 13 23:39:13.525905 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.525853    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"]
Jul 13 23:39:13.528193 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.528176    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.529092 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.529076    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=[]
Jul 13 23:39:13.529176 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.529096    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-jzqgh"]
Jul 13 23:39:13.529220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.529196    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.530448 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.530434    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-lwxsn"]
Jul 13 23:39:13.530548 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.530534    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.531782 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.531756    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-td2vq"]
Jul 13 23:39:13.531871 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.531838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.533039 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533022    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-s7blq"]
Jul 13 23:39:13.533151 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.533681 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533435    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xlffv\""
Jul 13 23:39:13.533681 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533533    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.533827 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533767    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.534046 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.533929    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 23:39:13.535257 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535093    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 23:39:13.535472 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wnhqt\""
Jul 13 23:39:13.535608 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535582    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.535711 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535673    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.535773 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535758    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 23:39:13.535998 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.535979    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-79hzk"]
Jul 13 23:39:13.536073 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.536036    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 23:39:13.536073 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.536051    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.536177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.536140    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 23:39:13.537545 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.537515    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 23:39:13.537545 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.537542    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.537700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.537585    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.537700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.537542    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vjqwz\""
Jul 13 23:39:13.538680 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.538664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vh992\""
Jul 13 23:39:13.538680 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.538675    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 23:39:13.538781 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.538675    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 23:39:13.539486 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.539464    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.542390 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542350    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 23:39:13.542481 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542412    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"]
Jul 13 23:39:13.542563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542509    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pj4hj\""
Jul 13 23:39:13.542563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.542563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542552    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 23:39:13.542794 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.542780    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.543833 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.543819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.546431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.546410    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.546431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.546427    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gs2f2\""
Jul 13 23:39:13.546562 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.546442    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.546562 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.546454    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 23:39:13.560727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.560710    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.560890 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.560877    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-lr4nq"]
Jul 13 23:39:13.562256 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.562240    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.564847 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.564832    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.564942 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.564929    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kvcdm\""
Jul 13 23:39:13.565009 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.564956    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.574860 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.574842    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"]
Jul 13 23:39:13.574860 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.574858    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 23:39:13.579728 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.579711    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.583494 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.583475    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6zqjf"]
Jul 13 23:39:13.585253 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.585238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.588389 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.588375    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 23:39:13.588510 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.588497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fnkhm\""
Jul 13 23:39:13.588659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.588647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 23:39:13.592978 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.592962    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pqr8z"]
Jul 13 23:39:13.593469 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.593456    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 23:39:13.594361 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.594347    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"]
Jul 13 23:39:13.594422 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.594412    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.594477 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.594461    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:13.601784 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.601768    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 23:39:13.604234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-multus-certs\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.604304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.604304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cnibin\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.604304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-device-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.604304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-k8s-cni-cncf-io\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-kubelet\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-sys\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-netd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adde175b-6b8b-4af9-b693-c6a2b9337104-iptables-alerter-script\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.604503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-var-lib-kubelet\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-socket-dir-parent\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-hostroot\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-systemd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a434d4f3-48e1-41f7-b82e-5481913fe081-host\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-env-overrides\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adde175b-6b8b-4af9-b693-c6a2b9337104-host-slash\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/650485c6eb65fd2e734a511b4a97894e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-174.ec2.internal\" (UID: \"650485c6eb65fd2e734a511b4a97894e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.604822 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-cni-binary-copy\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-ovn\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysconfig\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-log-socket\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-agent-certs\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-multus\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.604991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-systemd\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-host\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7bb\" (UniqueName: \"kubernetes.io/projected/a434d4f3-48e1-41f7-b82e-5481913fe081-kube-api-access-4b7bb\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605092    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-bin\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vft9l\" (UniqueName: \"kubernetes.io/projected/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-kube-api-access-vft9l\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqvn\" (UniqueName: \"kubernetes.io/projected/e6cad8c8-84e9-42d8-b267-57e1201c00d4-kube-api-access-pxqvn\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-cnibin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-run\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-tmp\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-netns\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-script-lib\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6g98\" (UniqueName: \"kubernetes.io/projected/adde175b-6b8b-4af9-b693-c6a2b9337104-kube-api-access-s6g98\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-sys-fs\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605578    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mqz7\" (UniqueName: \"kubernetes.io/projected/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kube-api-access-4mqz7\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-tuned\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr6g\" (UniqueName: \"kubernetes.io/projected/26c1f071-28f4-4448-9c76-bddfc89bf893-kube-api-access-smr6g\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-etc-kubernetes\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-var-lib-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-registration-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.605924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-system-cni-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.605991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-konnectivity-ca\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgvk\" (UniqueName: \"kubernetes.io/projected/125b0bad-f492-43c8-9846-41efc776bb2d-kube-api-access-5lgvk\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-modprobe-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606118    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-slash\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-config\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-multus-daemon-config\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a434d4f3-48e1-41f7-b82e-5481913fe081-serviceca\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-kubelet\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-systemd-units\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-node-log\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-conf\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmxk\" (UniqueName: \"kubernetes.io/projected/d3141d55-9c74-4ffc-a221-bdb2255c92ad-kube-api-access-nqmxk\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.606623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-os-release\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606526    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-lib-modules\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-os-release\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-bin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-conf-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-kubernetes\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-etc-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606701    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-socket-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-system-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.607388 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.606796    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-netns\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.612392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.612371    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-cjj8j"]
Jul 13 23:39:13.614387 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.614373    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:13.614433 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.614424    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:13.621555 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.621535    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-6mmj6"]
Jul 13 23:39:13.623241 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.623222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.625768 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.625748    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 23:39:13.625863 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.625770    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 23:39:13.626062 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.626039    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r9wgm\""
Jul 13 23:39:13.633889 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.633874    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9fw4w"]
Jul 13 23:39:13.635329 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.635315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.635382 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.635369    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:13.707307 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-agent-certs\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.707392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.707392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-multus\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.707392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd7c53-c383-4b60-a62b-82a34fbcc221-tmp-dir\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-multus\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-dbus\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-systemd\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-host\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7bb\" (UniqueName: \"kubernetes.io/projected/a434d4f3-48e1-41f7-b82e-5481913fe081-kube-api-access-4b7bb\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707572    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-host\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707587    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-systemd\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-bin\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vft9l\" (UniqueName: \"kubernetes.io/projected/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-kube-api-access-vft9l\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqvn\" (UniqueName: \"kubernetes.io/projected/e6cad8c8-84e9-42d8-b267-57e1201c00d4-kube-api-access-pxqvn\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707709    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-cnibin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707723    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-run\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-tmp\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-cnibin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-bin\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-netns\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.707813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-run\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-netns\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-script-lib\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6g98\" (UniqueName: \"kubernetes.io/projected/adde175b-6b8b-4af9-b693-c6a2b9337104-kube-api-access-s6g98\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-sys-fs\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.707989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mqz7\" (UniqueName: \"kubernetes.io/projected/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kube-api-access-4mqz7\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-tuned\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr6g\" (UniqueName: \"kubernetes.io/projected/26c1f071-28f4-4448-9c76-bddfc89bf893-kube-api-access-smr6g\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-etc-kubernetes\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708071    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-var-lib-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-registration-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708099    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708108    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-system-cni-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-konnectivity-ca\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgvk\" (UniqueName: \"kubernetes.io/projected/125b0bad-f492-43c8-9846-41efc776bb2d-kube-api-access-5lgvk\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.708381 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-modprobe-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-slash\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-config\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-registration-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708307    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-multus-daemon-config\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-var-lib-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a434d4f3-48e1-41f7-b82e-5481913fe081-serviceca\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-kubelet\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-systemd-units\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-modprobe-d\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-node-log\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48fd7c53-c383-4b60-a62b-82a34fbcc221-hosts-file\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-system-cni-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-etc-kubernetes\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.709261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-conf\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmxk\" (UniqueName: \"kubernetes.io/projected/d3141d55-9c74-4ffc-a221-bdb2255c92ad-kube-api-access-nqmxk\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708686    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-os-release\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-script-lib\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-lib-modules\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-os-release\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708800    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-bin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708811    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-kubelet\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-conf-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-kubernetes\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-node-log\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-systemd-units\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-etc-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-slash\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-socket-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.708980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-sys-fs\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.709013    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:13.710173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-socket-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.709077    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:14.209056066 +0000 UTC m=+2.496274394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-system-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-netns\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-multus-certs\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cnibin\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrt5k\" (UniqueName: \"kubernetes.io/projected/48fd7c53-c383-4b60-a62b-82a34fbcc221-kube-api-access-xrt5k\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-kubelet-config\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-device-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-k8s-cni-cncf-io\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-kubelet\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-sys\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-netd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.711082 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a434d4f3-48e1-41f7-b82e-5481913fe081-serviceca\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adde175b-6b8b-4af9-b693-c6a2b9337104-iptables-alerter-script\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-var-lib-kubelet\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovnkube-config\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-socket-dir-parent\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-hostroot\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-systemd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a434d4f3-48e1-41f7-b82e-5481913fe081-host\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-etc-selinux\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-system-cni-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-env-overrides\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adde175b-6b8b-4af9-b693-c6a2b9337104-host-slash\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-netns\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adde175b-6b8b-4af9-b693-c6a2b9337104-host-slash\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.711865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cnibin\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/650485c6eb65fd2e734a511b4a97894e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-174.ec2.internal\" (UID: \"650485c6eb65fd2e734a511b4a97894e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e6cad8c8-84e9-42d8-b267-57e1201c00d4-os-release\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-cni-binary-copy\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-device-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.709999    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-os-release\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/650485c6eb65fd2e734a511b4a97894e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-174.ec2.internal\" (UID: \"650485c6eb65fd2e734a511b4a97894e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710014    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-k8s-cni-cncf-io\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-multus-daemon-config\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710045    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-cni-bin\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-var-lib-kubelet\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-kubernetes\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-host-run-multus-certs\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-lib-modules\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.711387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.712725 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.710056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysctl-conf\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.711815    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-conf-dir\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-var-lib-kubelet\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-sys\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-etc-openvswitch\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712430    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-ovn-node-metrics-cert\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125b0bad-f492-43c8-9846-41efc776bb2d-cni-binary-copy\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-cni-netd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-ovn\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysconfig\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-systemd\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-log-socket\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712605    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-hostroot\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a434d4f3-48e1-41f7-b82e-5481913fe081-host\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-log-socket\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.713641 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-sysconfig\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3dab0b06682e13db962ddd68e270c56e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal\" (UID: \"3dab0b06682e13db962ddd68e270c56e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.712831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-run-ovn\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713079    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adde175b-6b8b-4af9-b693-c6a2b9337104-iptables-alerter-script\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713228    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-env-overrides\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-tmp\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e6cad8c8-84e9-42d8-b267-57e1201c00d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713718    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-agent-certs\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.713789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3141d55-9c74-4ffc-a221-bdb2255c92ad-etc-tuned\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.714580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.714138    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f93aec-e9a2-4864-a8f9-30631d8a7a4b-konnectivity-ca\") pod \"konnectivity-agent-s7blq\" (UID: \"27f93aec-e9a2-4864-a8f9-30631d8a7a4b\") " pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.715109 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.715060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/125b0bad-f492-43c8-9846-41efc776bb2d-multus-socket-dir-parent\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.717480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.717371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7bb\" (UniqueName: \"kubernetes.io/projected/a434d4f3-48e1-41f7-b82e-5481913fe081-kube-api-access-4b7bb\") pod \"node-ca-jzqgh\" (UID: \"a434d4f3-48e1-41f7-b82e-5481913fe081\") " pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.717701 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.717682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vft9l\" (UniqueName: \"kubernetes.io/projected/a1d7010e-7e90-481b-84ea-91a9b2c59b0f-kube-api-access-vft9l\") pod \"ovnkube-node-lwxsn\" (UID: \"a1d7010e-7e90-481b-84ea-91a9b2c59b0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.717786 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.717703    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqvn\" (UniqueName: \"kubernetes.io/projected/e6cad8c8-84e9-42d8-b267-57e1201c00d4-kube-api-access-pxqvn\") pod \"multus-additional-cni-plugins-6zqjf\" (UID: \"e6cad8c8-84e9-42d8-b267-57e1201c00d4\") " pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.717786 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.717746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6g98\" (UniqueName: \"kubernetes.io/projected/adde175b-6b8b-4af9-b693-c6a2b9337104-kube-api-access-s6g98\") pod \"iptables-alerter-td2vq\" (UID: \"adde175b-6b8b-4af9-b693-c6a2b9337104\") " pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.718824 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.718806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mqz7\" (UniqueName: \"kubernetes.io/projected/eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0-kube-api-access-4mqz7\") pod \"aws-ebs-csi-driver-node-d74r9\" (UID: \"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:13.719025 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.719004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr6g\" (UniqueName: \"kubernetes.io/projected/26c1f071-28f4-4448-9c76-bddfc89bf893-kube-api-access-smr6g\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:13.723520 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.723497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgvk\" (UniqueName: \"kubernetes.io/projected/125b0bad-f492-43c8-9846-41efc776bb2d-kube-api-access-5lgvk\") pod \"multus-79hzk\" (UID: \"125b0bad-f492-43c8-9846-41efc776bb2d\") " pod="openshift-multus/multus-79hzk"
Jul 13 23:39:13.723602 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.723589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmxk\" (UniqueName: \"kubernetes.io/projected/d3141d55-9c74-4ffc-a221-bdb2255c92ad-kube-api-access-nqmxk\") pod \"tuned-lr4nq\" (UID: \"d3141d55-9c74-4ffc-a221-bdb2255c92ad\") " pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.730713 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.730698    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lr4nq"
Jul 13 23:39:13.747612 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.747593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zqjf"
Jul 13 23:39:13.813059 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.813155 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd7c53-c383-4b60-a62b-82a34fbcc221-tmp-dir\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.813155 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-dbus\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.813155 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.813164    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.813228    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:14.313206164 +0000 UTC m=+2.600424490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-dbus\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48fd7c53-c383-4b60-a62b-82a34fbcc221-hosts-file\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813403    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48fd7c53-c383-4b60-a62b-82a34fbcc221-hosts-file\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrt5k\" (UniqueName: \"kubernetes.io/projected/48fd7c53-c383-4b60-a62b-82a34fbcc221-kube-api-access-xrt5k\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-kubelet-config\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.813484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48fd7c53-c383-4b60-a62b-82a34fbcc221-tmp-dir\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.813774 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.813518    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/caccffa7-8996-48b5-8eb1-1e83de888826-kubelet-config\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:13.819179 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.819163    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:13.819219 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.819182    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:13.819219 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.819192    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:13.819286 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:13.819249    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:14.319232975 +0000 UTC m=+2.606451302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:13.822184 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.822164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrt5k\" (UniqueName: \"kubernetes.io/projected/48fd7c53-c383-4b60-a62b-82a34fbcc221-kube-api-access-xrt5k\") pod \"node-resolver-6mmj6\" (UID: \"48fd7c53-c383-4b60-a62b-82a34fbcc221\") " pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:13.878587 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.878547    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.897108 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.897085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal"
Jul 13 23:39:13.906296 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.906268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jzqgh"
Jul 13 23:39:13.927177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.927157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:13.949369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.949323    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-td2vq"
Jul 13 23:39:13.962113 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.962095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:13.982977 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:13.982961    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-79hzk"
Jul 13 23:39:14.000040 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.000001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9"
Jul 13 23:39:14.053601 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.053578    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mmj6"
Jul 13 23:39:14.216134 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.216065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:14.216327 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.216241    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:14.216327 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.216316    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:15.216295297 +0000 UTC m=+3.503513626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:14.299524 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.299467    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 23:39:14.309910 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.309881    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 23:39:14.316504 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.316482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:14.316597 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.316578    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:14.316639 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.316628    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:15.31661205 +0000 UTC m=+3.603830374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:14.337130 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.337108    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmkkf"
Jul 13 23:39:14.344078 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.344062    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmkkf"
Jul 13 23:39:14.363827 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.363802    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cad8c8_84e9_42d8_b267_57e1201c00d4.slice/crio-6cc9d9102e19acd6611d4d2efd178beac0529bfff867a17bb1206fccf413371f WatchSource:0}: Error finding container 6cc9d9102e19acd6611d4d2efd178beac0529bfff867a17bb1206fccf413371f: Status 404 returned error can't find the container with id 6cc9d9102e19acd6611d4d2efd178beac0529bfff867a17bb1206fccf413371f
Jul 13 23:39:14.364384 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.364364    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dab0b06682e13db962ddd68e270c56e.slice/crio-463a9dc604bef1db6cf3bafc869ed78be7622c2b3b26f69cbf5db14f0593f7a8 WatchSource:0}: Error finding container 463a9dc604bef1db6cf3bafc869ed78be7622c2b3b26f69cbf5db14f0593f7a8: Status 404 returned error can't find the container with id 463a9dc604bef1db6cf3bafc869ed78be7622c2b3b26f69cbf5db14f0593f7a8
Jul 13 23:39:14.368001 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.367971    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3141d55_9c74_4ffc_a221_bdb2255c92ad.slice/crio-94800a007cd3aeb7648be2eb64d170a40c559b93a5525e29d6d687c79ed27ac3 WatchSource:0}: Error finding container 94800a007cd3aeb7648be2eb64d170a40c559b93a5525e29d6d687c79ed27ac3: Status 404 returned error can't find the container with id 94800a007cd3aeb7648be2eb64d170a40c559b93a5525e29d6d687c79ed27ac3
Jul 13 23:39:14.368792 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.368762    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab451f6_7cfe_45e2_ae3e_3ebf9ad908a0.slice/crio-bdb98b2ac52a4a5b3b7381f5c901ae9c679b8ac43cada133c96a15e0a6b3a58b WatchSource:0}: Error finding container bdb98b2ac52a4a5b3b7381f5c901ae9c679b8ac43cada133c96a15e0a6b3a58b: Status 404 returned error can't find the container with id bdb98b2ac52a4a5b3b7381f5c901ae9c679b8ac43cada133c96a15e0a6b3a58b
Jul 13 23:39:14.369483 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.369464    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125b0bad_f492_43c8_9846_41efc776bb2d.slice/crio-68adaf09ee3e442d17a07c1af100089c58bf1b21e81ab90dc923f38ce77cee66 WatchSource:0}: Error finding container 68adaf09ee3e442d17a07c1af100089c58bf1b21e81ab90dc923f38ce77cee66: Status 404 returned error can't find the container with id 68adaf09ee3e442d17a07c1af100089c58bf1b21e81ab90dc923f38ce77cee66
Jul 13 23:39:14.371314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.371137    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 23:39:14.405651 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.405616    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda434d4f3_48e1_41f7_b82e_5481913fe081.slice/crio-da63f9457eabc2df41415b1b2c748f3f3fabfdc8667a43be478996623effde5f WatchSource:0}: Error finding container da63f9457eabc2df41415b1b2c748f3f3fabfdc8667a43be478996623effde5f: Status 404 returned error can't find the container with id da63f9457eabc2df41415b1b2c748f3f3fabfdc8667a43be478996623effde5f
Jul 13 23:39:14.409073 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.409049    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadde175b_6b8b_4af9_b693_c6a2b9337104.slice/crio-0444cf762bd37d26193e8572d1299f76882bc0242cd1c7cdc0a80e096d926e43 WatchSource:0}: Error finding container 0444cf762bd37d26193e8572d1299f76882bc0242cd1c7cdc0a80e096d926e43: Status 404 returned error can't find the container with id 0444cf762bd37d26193e8572d1299f76882bc0242cd1c7cdc0a80e096d926e43
Jul 13 23:39:14.417611 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.417588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:14.417713 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.417689    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:14.417713 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.417700    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:14.417713 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.417711    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:14.418044 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.418030    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:15.41801297 +0000 UTC m=+3.705231296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:14.429166 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.429117    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerStarted","Data":"6cc9d9102e19acd6611d4d2efd178beac0529bfff867a17bb1206fccf413371f"}
Jul 13 23:39:14.430280 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.430256    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-td2vq" event={"ID":"adde175b-6b8b-4af9-b693-c6a2b9337104","Type":"ContainerStarted","Data":"0444cf762bd37d26193e8572d1299f76882bc0242cd1c7cdc0a80e096d926e43"}
Jul 13 23:39:14.431310 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.431289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzqgh" event={"ID":"a434d4f3-48e1-41f7-b82e-5481913fe081","Type":"ContainerStarted","Data":"da63f9457eabc2df41415b1b2c748f3f3fabfdc8667a43be478996623effde5f"}
Jul 13 23:39:14.432275 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.432247    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650485c6eb65fd2e734a511b4a97894e.slice/crio-137bbad71e089b67a14d70eec72a28fe3645f5afbe0b27ca7e9dd5091aef4df8 WatchSource:0}: Error finding container 137bbad71e089b67a14d70eec72a28fe3645f5afbe0b27ca7e9dd5091aef4df8: Status 404 returned error can't find the container with id 137bbad71e089b67a14d70eec72a28fe3645f5afbe0b27ca7e9dd5091aef4df8
Jul 13 23:39:14.432432 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.432412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-79hzk" event={"ID":"125b0bad-f492-43c8-9846-41efc776bb2d","Type":"ContainerStarted","Data":"68adaf09ee3e442d17a07c1af100089c58bf1b21e81ab90dc923f38ce77cee66"}
Jul 13 23:39:14.433647 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.433628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9" event={"ID":"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0","Type":"ContainerStarted","Data":"bdb98b2ac52a4a5b3b7381f5c901ae9c679b8ac43cada133c96a15e0a6b3a58b"}
Jul 13 23:39:14.434747 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.434726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lr4nq" event={"ID":"d3141d55-9c74-4ffc-a221-bdb2255c92ad","Type":"ContainerStarted","Data":"94800a007cd3aeb7648be2eb64d170a40c559b93a5525e29d6d687c79ed27ac3"}
Jul 13 23:39:14.435653 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:14.435634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal" event={"ID":"3dab0b06682e13db962ddd68e270c56e","Type":"ContainerStarted","Data":"463a9dc604bef1db6cf3bafc869ed78be7622c2b3b26f69cbf5db14f0593f7a8"}
Jul 13 23:39:14.462549 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.462528    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f93aec_e9a2_4864_a8f9_30631d8a7a4b.slice/crio-8120009e8f503a903718a27f4f92fae1d8cdb44b986cc573d625515b40a63dec WatchSource:0}: Error finding container 8120009e8f503a903718a27f4f92fae1d8cdb44b986cc573d625515b40a63dec: Status 404 returned error can't find the container with id 8120009e8f503a903718a27f4f92fae1d8cdb44b986cc573d625515b40a63dec
Jul 13 23:39:14.527788 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.527764    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fd7c53_c383_4b60_a62b_82a34fbcc221.slice/crio-81cac375fc44799b16d9d21323bf86ac06aa278d1a61129ab73b9915fb8ea3cc WatchSource:0}: Error finding container 81cac375fc44799b16d9d21323bf86ac06aa278d1a61129ab73b9915fb8ea3cc: Status 404 returned error can't find the container with id 81cac375fc44799b16d9d21323bf86ac06aa278d1a61129ab73b9915fb8ea3cc
Jul 13 23:39:14.528956 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:14.528936    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d7010e_7e90_481b_84ea_91a9b2c59b0f.slice/crio-95087a6b4b046bd2432faffb8e74daa070a62ab3c3e3f167afed6d228e6c18a4 WatchSource:0}: Error finding container 95087a6b4b046bd2432faffb8e74daa070a62ab3c3e3f167afed6d228e6c18a4: Status 404 returned error can't find the container with id 95087a6b4b046bd2432faffb8e74daa070a62ab3c3e3f167afed6d228e6c18a4
Jul 13 23:39:14.530943 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.530923    2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-lwxsn_openshift-ovn-kubernetes(a1d7010e-7e90-481b-84ea-91a9b2c59b0f): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 13 23:39:14.532565 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.532544    2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-lwxsn_openshift-ovn-kubernetes(a1d7010e-7e90-481b-84ea-91a9b2c59b0f): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 13 23:39:14.537755 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:14.537711    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" podUID="a1d7010e-7e90-481b-84ea-91a9b2c59b0f"
Jul 13 23:39:15.223671 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.223537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:15.223863 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.223718    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:15.223863 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.223780    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:17.22376109 +0000 UTC m=+5.510979421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:15.324500 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.324463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:15.325323 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.324634    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:15.325323 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.324693    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:17.324674387 +0000 UTC m=+5.611892718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:15.347162 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.347106    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 23:34:14 +0000 UTC" deadline="2028-01-29 00:38:55.110675633 +0000 UTC"
Jul 13 23:39:15.347162 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.347154    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13536h59m39.763526206s"
Jul 13 23:39:15.425535 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.425497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:15.425761 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.425725    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:15.425761 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.425750    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:15.425893 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.425764    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:15.425893 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.425825    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:17.425805054 +0000 UTC m=+5.713023422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.428656    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.428784    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.428877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.428990    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.429047    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:15.429274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.429132    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:15.454696 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.454653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"95087a6b4b046bd2432faffb8e74daa070a62ab3c3e3f167afed6d228e6c18a4"}
Jul 13 23:39:15.484809 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.484737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mmj6" event={"ID":"48fd7c53-c383-4b60-a62b-82a34fbcc221","Type":"ContainerStarted","Data":"81cac375fc44799b16d9d21323bf86ac06aa278d1a61129ab73b9915fb8ea3cc"}
Jul 13 23:39:15.497149 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:15.497077    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fc4c36822d1ce0f4acab78208d171b757bca1285f69c9653d5a959fa853d8470\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" podUID="a1d7010e-7e90-481b-84ea-91a9b2c59b0f"
Jul 13 23:39:15.497694 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.497658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7blq" event={"ID":"27f93aec-e9a2-4864-a8f9-30631d8a7a4b","Type":"ContainerStarted","Data":"8120009e8f503a903718a27f4f92fae1d8cdb44b986cc573d625515b40a63dec"}
Jul 13 23:39:15.528828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:15.528669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal" event={"ID":"650485c6eb65fd2e734a511b4a97894e","Type":"ContainerStarted","Data":"137bbad71e089b67a14d70eec72a28fe3645f5afbe0b27ca7e9dd5091aef4df8"}
Jul 13 23:39:16.347845 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:16.347441    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 23:34:14 +0000 UTC" deadline="2028-01-17 20:35:28.551282185 +0000 UTC"
Jul 13 23:39:16.347845 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:16.347506    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13268h56m12.203779943s"
Jul 13 23:39:17.241892 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.241827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:17.242090 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.241990    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:17.242090 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.242061    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:21.242039921 +0000 UTC m=+9.529258263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:17.343770 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.343163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:17.343770 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.343346    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:17.343770 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.343413    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:21.343392149 +0000 UTC m=+9.630610493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.426482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.426491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.426635    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.426713    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.426779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:17.426999 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.426861    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:17.444197 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:17.443831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:17.444197 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.444081    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:17.444197 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.444102    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:17.444197 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.444115    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:17.444197 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:17.444166    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:21.444147425 +0000 UTC m=+9.731365753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:19.426417 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:19.426381    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:19.426417 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:19.426404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:19.426417 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:19.426420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:19.427078 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:19.426539    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:19.427078 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:19.426599    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:19.427078 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:19.426668    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:21.275479 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.275442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:21.275928 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.275593    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:21.275928 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.275670    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:29.275647181 +0000 UTC m=+17.562865511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:21.376393 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.376358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:21.376549 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.376528    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:21.376622 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.376600    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:29.37657815 +0000 UTC m=+17.663796497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:21.427015 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.426981    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:21.427180 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.427018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:21.427180 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.427025    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:21.427180 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.427119    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:21.427533 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.427505    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:21.427618 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.427595    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:21.477157 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:21.477125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:21.477310 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.477293    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:21.477376 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.477312    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:21.477376 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.477324    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:21.477474 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:21.477388    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:29.477370435 +0000 UTC m=+17.764588774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:23.426745 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:23.426709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:23.427244 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:23.426819    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:23.427244 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:23.426827    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:23.427244 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:23.426855    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:23.427244 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:23.426919    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:23.427244 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:23.426972    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:25.426454 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:25.426418    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:25.426954 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:25.426532    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:25.426954 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:25.426418    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:25.426954 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:25.426563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:25.426954 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:25.426653    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:25.426954 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:25.426753    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:27.425904 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:27.425858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:27.426365 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:27.425858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:27.426365 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:27.425979    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:27.426365 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:27.425994    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:27.426365 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:27.426056    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:27.426365 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:27.426100    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:28.563757 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.563451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lr4nq" event={"ID":"d3141d55-9c74-4ffc-a221-bdb2255c92ad","Type":"ContainerStarted","Data":"27030d2aa7f030c8f95b41c37c350f6589d3ad630ce6c6e1385fcf525853e466"}
Jul 13 23:39:28.565643 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.565618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal" event={"ID":"650485c6eb65fd2e734a511b4a97894e","Type":"ContainerStarted","Data":"07af5ce1b436bfb151ec5e94af9cff91b392b275be34d648f37544ec9ff6282e"}
Jul 13 23:39:28.572671 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.572621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-79hzk" event={"ID":"125b0bad-f492-43c8-9846-41efc776bb2d","Type":"ContainerStarted","Data":"59f82042e2eb6e900ddddd08c4ea0f376c32975804f113a57466e6ee9bcad448"}
Jul 13 23:39:28.580062 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.580001    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lr4nq" podStartSLOduration=0.940327941 podStartE2EDuration="14.579985493s" totalImagesPullingTime="13.639657552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:28.579976861 +0000 UTC m=+16.867195209" watchObservedRunningTime="2026-07-13 23:39:28.579985493 +0000 UTC m=+16.867203842"
Jul 13 23:39:28.595449 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.595399    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-79hzk" podStartSLOduration=0.929517996 podStartE2EDuration="14.59537991s" totalImagesPullingTime="13.665861914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:28.595194315 +0000 UTC m=+16.882412662" watchObservedRunningTime="2026-07-13 23:39:28.59537991 +0000 UTC m=+16.882598257"
Jul 13 23:39:28.613851 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:28.613796    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-174.ec2.internal" podStartSLOduration=14.613781713 podStartE2EDuration="14.613781713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:28.613652622 +0000 UTC m=+16.900870969" watchObservedRunningTime="2026-07-13 23:39:28.613781713 +0000 UTC m=+16.901000058"
Jul 13 23:39:29.336149 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.335834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:29.336313 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.336021    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:29.336313 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.336257    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:45.3362341 +0000 UTC m=+33.623452433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:29.426926 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.426881    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:29.427095 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.426881    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:29.427095 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.427012    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:29.427095 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.427031    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:29.427255 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.427118    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:29.427255 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.427204    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:29.436949 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.436881    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:29.437104 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.437010    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:29.437104 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.437079    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:45.437058839 +0000 UTC m=+33.724277187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:29.537366 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.537327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:29.537544 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.537508    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:29.537544 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.537535    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:29.537645 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.537548    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:29.537645 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:29.537614    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:45.537592886 +0000 UTC m=+33.824811223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:29.576852 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.576622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s7blq" event={"ID":"27f93aec-e9a2-4864-a8f9-30631d8a7a4b","Type":"ContainerStarted","Data":"64a391cc6639bd1379a5ce3584c99da1e8ad4f9066ee2af49b7208ef0406227e"}
Jul 13 23:39:29.578314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.578287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jzqgh" event={"ID":"a434d4f3-48e1-41f7-b82e-5481913fe081","Type":"ContainerStarted","Data":"ed3e5f74f3bfa82945bf1b3e09af961b9bf130a1a585af322a0532a2914c8347"}
Jul 13 23:39:29.580041 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.580015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9" event={"ID":"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0","Type":"ContainerStarted","Data":"427abeca218f843b3ca654f3efc1d93fbdfd0a94ec89f4f76e55b080eb420f45"}
Jul 13 23:39:29.581798 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.581773    2558 generic.go:356] "Generic (PLEG): container finished" podID="3dab0b06682e13db962ddd68e270c56e" containerID="fe649045da1fc71b9aa9a5b697388ac7f0cb1c5692d8a2c41bbea7c672010459" exitCode=0
Jul 13 23:39:29.581977 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.581957    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal" event={"ID":"3dab0b06682e13db962ddd68e270c56e","Type":"ContainerDied","Data":"fe649045da1fc71b9aa9a5b697388ac7f0cb1c5692d8a2c41bbea7c672010459"}
Jul 13 23:39:29.583593 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.583564    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="41436a070c894005feceb837077d33c6c202f56b956dfcbd59cd6432fad57d1d" exitCode=0
Jul 13 23:39:29.583683 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.583645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"41436a070c894005feceb837077d33c6c202f56b956dfcbd59cd6432fad57d1d"}
Jul 13 23:39:29.585637 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.585609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mmj6" event={"ID":"48fd7c53-c383-4b60-a62b-82a34fbcc221","Type":"ContainerStarted","Data":"140e4fbebef28884c5d3b0d5850ffaa10257a731e534d71479a43d82a2853881"}
Jul 13 23:39:29.590931 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.590880    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s7blq" podStartSLOduration=2.042826733 podStartE2EDuration="15.590868915s" totalImagesPullingTime="13.548042182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:29.590779645 +0000 UTC m=+17.877998002" watchObservedRunningTime="2026-07-13 23:39:29.590868915 +0000 UTC m=+17.878087261"
Jul 13 23:39:29.640421 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.640365    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jzqgh" podStartSLOduration=2.036601048 podStartE2EDuration="15.640344818s" totalImagesPullingTime="13.60374377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:29.618941339 +0000 UTC m=+17.906159688" watchObservedRunningTime="2026-07-13 23:39:29.640344818 +0000 UTC m=+17.927563173"
Jul 13 23:39:29.657588 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:29.657535    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6mmj6" podStartSLOduration=2.162959518 podStartE2EDuration="15.657513043s" totalImagesPullingTime="13.494553525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:29.657082102 +0000 UTC m=+17.944300448" watchObservedRunningTime="2026-07-13 23:39:29.657513043 +0000 UTC m=+17.944731390"
Jul 13 23:39:30.590530 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.590061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-td2vq" event={"ID":"adde175b-6b8b-4af9-b693-c6a2b9337104","Type":"ContainerStarted","Data":"9e5a7c27f2ec1afb3855825b884e7f309903d1ee246045a09052186dfa26e28e"}
Jul 13 23:39:30.593634 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.593563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal" event={"ID":"3dab0b06682e13db962ddd68e270c56e","Type":"ContainerStarted","Data":"6de101b474479ddb1da177b2f0e2124a3741ceae50f35eb5ace388d500feadd9"}
Jul 13 23:39:30.605177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.605116    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-td2vq" podStartSLOduration=2.984145333 podStartE2EDuration="16.605095558s" totalImagesPullingTime="13.620950225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:30.60434423 +0000 UTC m=+18.891562576" watchObservedRunningTime="2026-07-13 23:39:30.605095558 +0000 UTC m=+18.892313905"
Jul 13 23:39:30.614007 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.613972    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 23:39:30.618660 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.618607    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-174.ec2.internal" podStartSLOduration=16.618593859 podStartE2EDuration="16.618593859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:30.618566231 +0000 UTC m=+18.905784578" watchObservedRunningTime="2026-07-13 23:39:30.618593859 +0000 UTC m=+18.905812205"
Jul 13 23:39:30.839792 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.839711    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:30.840478 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:30.840408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:31.354138 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.354016    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T23:39:30.613991726Z","UUID":"9e9b8360-e96e-4c47-b513-455ae8ff217f","Handler":null,"Name":"","Endpoint":""}
Jul 13 23:39:31.355879 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.355852    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 23:39:31.356062 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.355920    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 23:39:31.426357 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.426326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:31.426525 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:31.426462    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:31.426525 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.426495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:31.426637 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:31.426576    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:31.426684 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.426638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:31.426727 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:31.426706    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:31.598807 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:31.598718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9" event={"ID":"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0","Type":"ContainerStarted","Data":"33bd49d3c08f46a393aad0448bb5e93b1562d8dc3293c6d3b0de864f622eb84f"}
Jul 13 23:39:32.603500 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:32.603474    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 13 23:39:32.603500 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:32.603478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9" event={"ID":"eab451f6-7cfe-45e2-ae3e-3ebf9ad908a0","Type":"ContainerStarted","Data":"0f18b8f90dc6577f408e90f2359e2cad9d82319f1fbf93427887853f72b68aee"}
Jul 13 23:39:32.620597 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:32.620552    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d74r9" podStartSLOduration=2.339677192 podStartE2EDuration="18.620537037s" totalImagesPullingTime="16.280859845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:32.620095414 +0000 UTC m=+20.907313750" watchObservedRunningTime="2026-07-13 23:39:32.620537037 +0000 UTC m=+20.907755380"
Jul 13 23:39:33.427755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:33.427535    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:33.427969 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:33.427861    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:33.427969 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:33.427612    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:33.428127 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:33.427968    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:33.428127 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:33.427586    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:33.428127 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:33.428039    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:35.426180 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:35.426144    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:35.426828 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:35.426307    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:35.426828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:35.426361    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:35.426828 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:35.426425    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:35.426828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:35.426473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:35.426828 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:35.426576    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:37.426858 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:37.426818    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:37.427342 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:37.426818    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:37.427342 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:37.426821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:37.427342 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:37.426982    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:37.427342 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:37.427087    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:37.427342 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:37.427148    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:38.327279 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:38.327247    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:38.327511 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:38.327422    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 13 23:39:38.328164 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:38.328140    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s7blq"
Jul 13 23:39:39.426962 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.426775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:39.427414 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.426794    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:39.427414 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:39.427031    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:39.427414 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.426800    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:39.427414 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:39.427104    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:39.427414 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:39.427212    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:39.619209 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"e0179ea124bbd8d74a43ef1b00a60b8fb4283a8d385a635bef54b471759be356"}
Jul 13 23:39:39.619341 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"783c85310374c508ffaff1e1c3832be69f0adde65d0862b1cd4b80bf7eaa15d8"}
Jul 13 23:39:39.619341 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"539440d84ffe44ac3c7a2eecad7356a3a21470a0d7316ebf53073e60742e8031"}
Jul 13 23:39:39.619341 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619292    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"00fc820076dff49b957dcdbc8c235e108c54f78ccd2acc32a860e9c6c80b2978"}
Jul 13 23:39:39.619341 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"616d5e837af6a8b75e58062b76d1ac04d7c5fb15cc5be92e1c25aacdcef7fc3d"}
Jul 13 23:39:39.619500 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.619375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"37cf4fb91c3bf52577601f423e00580d31a3041be974140dec2b23ec9f9b0ea0"}
Jul 13 23:39:39.620877 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.620855    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="39b0300bae20c7e47a8928f4c23557cf5328f254b1c38b30e54fb9b1b15745d0" exitCode=0
Jul 13 23:39:39.620970 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:39.620899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"39b0300bae20c7e47a8928f4c23557cf5328f254b1c38b30e54fb9b1b15745d0"}
Jul 13 23:39:40.625040 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:40.625001    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="1d85ea1ef63dab4efd14a07346a9bd69f3135e23bfd11c14ce8f6686d313c55c" exitCode=0
Jul 13 23:39:40.625490 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:40.625057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"1d85ea1ef63dab4efd14a07346a9bd69f3135e23bfd11c14ce8f6686d313c55c"}
Jul 13 23:39:41.426048 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.425960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:41.426048 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.425976    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:41.426242 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:41.426078    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:41.426242 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:41.426155    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:41.426242 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.426172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:41.426337 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:41.426267    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:41.629492 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.629455    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="d03a619303c585563e6d3de580c50b0ab5a86781994fa55bf9b108b63fc63310" exitCode=0
Jul 13 23:39:41.629992 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.629536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"d03a619303c585563e6d3de580c50b0ab5a86781994fa55bf9b108b63fc63310"}
Jul 13 23:39:41.632857 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:41.632836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"45fcf78021848509e4cd8e00a11abbd6dd87e38c8b5b83ef0508bc6efd2ad67a"}
Jul 13 23:39:43.427523 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.427485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:43.427992 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.427571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:43.427992 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.427615    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:43.427992 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:43.427631    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:43.427992 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:43.427744    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:43.427992 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:43.427802    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:43.644538 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.644283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" event={"ID":"a1d7010e-7e90-481b-84ea-91a9b2c59b0f","Type":"ContainerStarted","Data":"c17e74c00c3b76367a59885f597990caf8a9e6c8676968392e3e7e7d06cd3eb5"}
Jul 13 23:39:43.644865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.644574    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:43.662300 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.662267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:43.674937 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:43.674748    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn" podStartSLOduration=5.243061419 podStartE2EDuration="29.674732185s" totalImagesPullingTime="24.431670766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-13 23:39:43.674398102 +0000 UTC m=+31.961616448" watchObservedRunningTime="2026-07-13 23:39:43.674732185 +0000 UTC m=+31.961950529"
Jul 13 23:39:44.646738 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:44.646705    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:44.647408 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:44.646788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:44.663003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:44.662975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:39:45.355407 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.355366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:45.355553 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.355530    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:45.355625 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.355614    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs podName:26c1f071-28f4-4448-9c76-bddfc89bf893 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:17.355592276 +0000 UTC m=+65.642810619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs") pod "network-metrics-daemon-pqr8z" (UID: "26c1f071-28f4-4448-9c76-bddfc89bf893") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 23:39:45.426234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.426204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:45.426431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.426255    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:45.426431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.426215    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:45.426431 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.426340    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:45.426624 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.426466    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:45.426624 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.426560    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:45.456531 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.456493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:45.456690 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.456641    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:45.456759 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.456710    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret podName:caccffa7-8996-48b5-8eb1-1e83de888826 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:17.456693393 +0000 UTC m=+65.743911716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret") pod "global-pull-secret-syncer-9fw4w" (UID: "caccffa7-8996-48b5-8eb1-1e83de888826") : object "kube-system"/"original-pull-secret" not registered
Jul 13 23:39:45.527874 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.527846    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fw4w"]
Jul 13 23:39:45.531577 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.531544    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjj8j"]
Jul 13 23:39:45.532279 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.532253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pqr8z"]
Jul 13 23:39:45.557099 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.557068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:45.557253 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.557235    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 23:39:45.557298 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.557258    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 23:39:45.557298 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.557267    2558 projected.go:196] Error preparing data for projected volume kube-api-access-qvt42 for pod openshift-network-diagnostics/network-check-target-cjj8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:45.557362 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.557317    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42 podName:836a5748-117e-42ca-9b40-aed9fbbb4807 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:17.557304737 +0000 UTC m=+65.844523060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvt42" (UniqueName: "kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42") pod "network-check-target-cjj8j" (UID: "836a5748-117e-42ca-9b40-aed9fbbb4807") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 23:39:45.647736 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.647713    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:45.648075 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.647718    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:45.648075 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:45.647725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:45.648075 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.647814    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:45.648075 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.647905    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:45.648295 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:45.648273    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:46.652598 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:46.652566    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="a670f0a37b48fe392d3b0141b8872985c7c7c8044147acac849e116f9d4f754b" exitCode=0
Jul 13 23:39:46.653110 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:46.652647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"a670f0a37b48fe392d3b0141b8872985c7c7c8044147acac849e116f9d4f754b"}
Jul 13 23:39:47.426735 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:47.426493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:47.426908 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:47.426496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:47.426908 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:47.426843    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:47.426908 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:47.426532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:47.427041 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:47.426908    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:47.427041 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:47.426987    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:47.658696 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:47.658662    2558 generic.go:356] "Generic (PLEG): container finished" podID="e6cad8c8-84e9-42d8-b267-57e1201c00d4" containerID="58eb9d6669c814244efb13aee72ce0887d525cdce0093203ef43a2487898d7b7" exitCode=0
Jul 13 23:39:47.659022 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:47.658722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerDied","Data":"58eb9d6669c814244efb13aee72ce0887d525cdce0093203ef43a2487898d7b7"}
Jul 13 23:39:48.664244 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:48.664198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" event={"ID":"e6cad8c8-84e9-42d8-b267-57e1201c00d4","Type":"ContainerStarted","Data":"3c5ac5d59ba2e6628a8dd4b39279088ff49770624861e9acb1a58f2bb78019d9"}
Jul 13 23:39:48.686788 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:48.686738    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zqjf" podStartSLOduration=6.467997866 podStartE2EDuration="34.68671867s" totalImagesPullingTime="28.218720804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:48.686141454 +0000 UTC m=+36.973359800" watchObservedRunningTime="2026-07-13 23:39:48.68671867 +0000 UTC m=+36.973937019"
Jul 13 23:39:49.426799 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:49.426757    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:49.426992 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:49.426767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:49.426992 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:49.426866    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:49.426992 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:49.426969    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:49.427171 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:49.426771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:49.427171 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:49.427079    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:51.426848 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.426774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:51.427274 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.426774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:51.427274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.426871    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjj8j" podUID="836a5748-117e-42ca-9b40-aed9fbbb4807"
Jul 13 23:39:51.427274 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.426781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:51.427274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.426979    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pqr8z" podUID="26c1f071-28f4-4448-9c76-bddfc89bf893"
Jul 13 23:39:51.427274 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.427048    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fw4w" podUID="caccffa7-8996-48b5-8eb1-1e83de888826"
Jul 13 23:39:51.506443 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.506416    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-174.ec2.internal" event="NodeReady"
Jul 13 23:39:51.506610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.506536    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 23:39:51.537983 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.537954    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:39:51.561855 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.561833    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kpnjw"]
Jul 13 23:39:51.562037 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.562016    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.564696 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.564668    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8vhjc\""
Jul 13 23:39:51.564808 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.564713    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 23:39:51.564808 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.564776    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 23:39:51.565112 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.565094    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 23:39:51.570255 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.570087    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 23:39:51.577816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.577798    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sxkvh"]
Jul 13 23:39:51.577934 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.577920    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.580412 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.580395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 23:39:51.580505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.580444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 23:39:51.580505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.580450    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b7hbx\""
Jul 13 23:39:51.580505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.580452    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 23:39:51.593208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.593190    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:39:51.593311 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.593215    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kpnjw"]
Jul 13 23:39:51.593311 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.593230    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxkvh"]
Jul 13 23:39:51.593424 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.593321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.596756 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.596736    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 23:39:51.596823 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.596804    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 23:39:51.596823 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.596815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zw85q\""
Jul 13 23:39:51.705228 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e73bef9-456c-4491-b77f-7b4e95f0b355-config-volume\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.705228 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705228 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ef6df1-688a-4170-ad90-4cdfdab2528c-cert\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wwx\" (UniqueName: \"kubernetes.io/projected/3e73bef9-456c-4491-b77f-7b4e95f0b355-kube-api-access-79wwx\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705480 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6p9\" (UniqueName: \"kubernetes.io/projected/99ef6df1-688a-4170-ad90-4cdfdab2528c-kube-api-access-pt6p9\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.705763 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705763 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705763 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzxr\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.705763 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.705666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e73bef9-456c-4491-b77f-7b4e95f0b355-tmp-dir\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.806496 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e73bef9-456c-4491-b77f-7b4e95f0b355-config-volume\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806538    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ef6df1-688a-4170-ad90-4cdfdab2528c-cert\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.806659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wwx\" (UniqueName: \"kubernetes.io/projected/3e73bef9-456c-4491-b77f-7b4e95f0b355-kube-api-access-79wwx\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.806674    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.806697    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bddfd4ff-wdnd5: secret "image-registry-tls" not found
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806699    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6p9\" (UniqueName: \"kubernetes.io/projected/99ef6df1-688a-4170-ad90-4cdfdab2528c-kube-api-access-pt6p9\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.806766    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls podName:1e8f7563-0a5d-4af9-a105-c1ee533ec1ea nodeName:}" failed. No retries permitted until 2026-07-13 23:39:52.306747801 +0000 UTC m=+40.593966139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls") pod "image-registry-64bddfd4ff-wdnd5" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea") : secret "image-registry-tls" not found
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzxr\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807003 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.806910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e73bef9-456c-4491-b77f-7b4e95f0b355-tmp-dir\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.807413 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.807070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807413 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.807163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e73bef9-456c-4491-b77f-7b4e95f0b355-tmp-dir\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.807413 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.807212    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e73bef9-456c-4491-b77f-7b4e95f0b355-config-volume\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.807413 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.807374    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 23:39:51.807604 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:51.807453    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls podName:3e73bef9-456c-4491-b77f-7b4e95f0b355 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:52.307418857 +0000 UTC m=+40.594637181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls") pod "dns-default-sxkvh" (UID: "3e73bef9-456c-4491-b77f-7b4e95f0b355") : secret "dns-default-metrics-tls" not found
Jul 13 23:39:51.807604 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.807509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.807715 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.807606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.810903 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.810856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.811028 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.810921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.811028 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.810943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99ef6df1-688a-4170-ad90-4cdfdab2528c-cert\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.815315 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.815274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.815688 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.815668    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6p9\" (UniqueName: \"kubernetes.io/projected/99ef6df1-688a-4170-ad90-4cdfdab2528c-kube-api-access-pt6p9\") pod \"ingress-canary-kpnjw\" (UID: \"99ef6df1-688a-4170-ad90-4cdfdab2528c\") " pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:51.815688 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.815679    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzxr\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:51.816468 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.816452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wwx\" (UniqueName: \"kubernetes.io/projected/3e73bef9-456c-4491-b77f-7b4e95f0b355-kube-api-access-79wwx\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:51.889234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:51.889199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kpnjw"
Jul 13 23:39:52.073244 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:52.073067    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kpnjw"]
Jul 13 23:39:52.076202 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:52.076172    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ef6df1_688a_4170_ad90_4cdfdab2528c.slice/crio-3adbbb01938d44c9da84b37098910eace0d360afd4684a6430ee99f11d493f00 WatchSource:0}: Error finding container 3adbbb01938d44c9da84b37098910eace0d360afd4684a6430ee99f11d493f00: Status 404 returned error can't find the container with id 3adbbb01938d44c9da84b37098910eace0d360afd4684a6430ee99f11d493f00
Jul 13 23:39:52.310010 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:52.309984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:52.310156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:52.310021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:52.310156 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:52.310143    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 23:39:52.310228 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:52.310156    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 23:39:52.310228 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:52.310161    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bddfd4ff-wdnd5: secret "image-registry-tls" not found
Jul 13 23:39:52.310228 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:52.310213    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls podName:1e8f7563-0a5d-4af9-a105-c1ee533ec1ea nodeName:}" failed. No retries permitted until 2026-07-13 23:39:53.310197915 +0000 UTC m=+41.597416239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls") pod "image-registry-64bddfd4ff-wdnd5" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea") : secret "image-registry-tls" not found
Jul 13 23:39:52.310228 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:52.310227    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls podName:3e73bef9-456c-4491-b77f-7b4e95f0b355 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:53.310222216 +0000 UTC m=+41.597440539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls") pod "dns-default-sxkvh" (UID: "3e73bef9-456c-4491-b77f-7b4e95f0b355") : secret "dns-default-metrics-tls" not found
Jul 13 23:39:52.675462 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:52.675376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kpnjw" event={"ID":"99ef6df1-688a-4170-ad90-4cdfdab2528c","Type":"ContainerStarted","Data":"3adbbb01938d44c9da84b37098910eace0d360afd4684a6430ee99f11d493f00"}
Jul 13 23:39:53.320770 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.320733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:53.320985 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.320780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:53.320985 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:53.320905    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 23:39:53.320985 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:53.320911    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 23:39:53.321140 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:53.320992    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bddfd4ff-wdnd5: secret "image-registry-tls" not found
Jul 13 23:39:53.321140 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:53.321000    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls podName:3e73bef9-456c-4491-b77f-7b4e95f0b355 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:55.320968258 +0000 UTC m=+43.608186597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls") pod "dns-default-sxkvh" (UID: "3e73bef9-456c-4491-b77f-7b4e95f0b355") : secret "dns-default-metrics-tls" not found
Jul 13 23:39:53.321140 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:53.321049    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls podName:1e8f7563-0a5d-4af9-a105-c1ee533ec1ea nodeName:}" failed. No retries permitted until 2026-07-13 23:39:55.321027654 +0000 UTC m=+43.608246000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls") pod "image-registry-64bddfd4ff-wdnd5" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea") : secret "image-registry-tls" not found
Jul 13 23:39:53.427593 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.427565    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:39:53.427792 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.427692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:39:53.427792 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.427708    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:39:53.430105 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.430081    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5s29d\""
Jul 13 23:39:53.430367 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.430353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 23:39:53.431417 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.431396    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 23:39:53.431525 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.431440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qp66b\""
Jul 13 23:39:53.431525 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.431474    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 23:39:53.431525 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.431497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 23:39:53.744078 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.743998    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7ds97"]
Jul 13 23:39:53.770129 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.770098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7ds97"]
Jul 13 23:39:53.770284 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.770229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:53.772869 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.772839    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 23:39:53.773752 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.773724    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 23:39:53.773833 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.773750    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 23:39:53.773833 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.773730    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9c6sz\""
Jul 13 23:39:53.773833 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.773792    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 23:39:53.926369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.926332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:53.926538 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.926398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77753122-f06d-44ff-a1d4-dc65e5524159-data-volume\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:53.926538 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.926454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:53.926538 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.926512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vfs\" (UniqueName: \"kubernetes.io/projected/77753122-f06d-44ff-a1d4-dc65e5524159-kube-api-access-v4vfs\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:53.926673 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:53.926572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77753122-f06d-44ff-a1d4-dc65e5524159-crio-socket\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.027790 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.027761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77753122-f06d-44ff-a1d4-dc65e5524159-crio-socket\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.027987 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.027815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.027987 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.027916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77753122-f06d-44ff-a1d4-dc65e5524159-data-volume\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.027987 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.027954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.027987 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.027963    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:54.028213 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.028008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vfs\" (UniqueName: \"kubernetes.io/projected/77753122-f06d-44ff-a1d4-dc65e5524159-kube-api-access-v4vfs\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.028213 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.028033    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:54.528005489 +0000 UTC m=+42.815223815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:54.028213 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.028056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77753122-f06d-44ff-a1d4-dc65e5524159-crio-socket\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.028213 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.028125    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 23:39:54.028395 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.028224    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:54.528204196 +0000 UTC m=+42.815422543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : configmap "kube-rbac-proxy" not found
Jul 13 23:39:54.028395 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.028309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77753122-f06d-44ff-a1d4-dc65e5524159-data-volume\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.041389 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.041371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vfs\" (UniqueName: \"kubernetes.io/projected/77753122-f06d-44ff-a1d4-dc65e5524159-kube-api-access-v4vfs\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.428938 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.428854    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"]
Jul 13 23:39:54.441182 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.441125    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"
Jul 13 23:39:54.442120 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.442100    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"]
Jul 13 23:39:54.443657 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.443638    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 13 23:39:54.444554 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.444535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 13 23:39:54.444678 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.444542    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zxpzj\""
Jul 13 23:39:54.532431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.532405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.532547 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.532435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmbx5\" (UniqueName: \"kubernetes.io/projected/7eb9e210-8467-458b-b364-437157f57a02-kube-api-access-bmbx5\") pod \"migrator-7f59b54f6c-z5wkq\" (UID: \"7eb9e210-8467-458b-b364-437157f57a02\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"
Jul 13 23:39:54.532547 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.532470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:54.532547 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.532543    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 23:39:54.532641 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.532582    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:54.532641 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.532586    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:55.532573205 +0000 UTC m=+43.819791529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : configmap "kube-rbac-proxy" not found
Jul 13 23:39:54.532707 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:54.532643    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:55.532625109 +0000 UTC m=+43.819843439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:54.633760 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.633731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmbx5\" (UniqueName: \"kubernetes.io/projected/7eb9e210-8467-458b-b364-437157f57a02-kube-api-access-bmbx5\") pod \"migrator-7f59b54f6c-z5wkq\" (UID: \"7eb9e210-8467-458b-b364-437157f57a02\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"
Jul 13 23:39:54.641290 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.641272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmbx5\" (UniqueName: \"kubernetes.io/projected/7eb9e210-8467-458b-b364-437157f57a02-kube-api-access-bmbx5\") pod \"migrator-7f59b54f6c-z5wkq\" (UID: \"7eb9e210-8467-458b-b364-437157f57a02\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"
Jul 13 23:39:54.750741 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.750720    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"
Jul 13 23:39:54.888610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:54.888578    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq"]
Jul 13 23:39:54.891750 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:54.891723    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb9e210_8467_458b_b364_437157f57a02.slice/crio-99d791a41432ec20f87bb3afd822ee630fc2c195b11b6ec24024c4af6ba92e71 WatchSource:0}: Error finding container 99d791a41432ec20f87bb3afd822ee630fc2c195b11b6ec24024c4af6ba92e71: Status 404 returned error can't find the container with id 99d791a41432ec20f87bb3afd822ee630fc2c195b11b6ec24024c4af6ba92e71
Jul 13 23:39:55.329041 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.328864    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mmj6_48fd7c53-c383-4b60-a62b-82a34fbcc221/dns-node-resolver/0.log"
Jul 13 23:39:55.339075 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.339027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:55.339154 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.339124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:55.339208 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.339162    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 23:39:55.339245 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.339215    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls podName:3e73bef9-456c-4491-b77f-7b4e95f0b355 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:59.339201212 +0000 UTC m=+47.626419535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls") pod "dns-default-sxkvh" (UID: "3e73bef9-456c-4491-b77f-7b4e95f0b355") : secret "dns-default-metrics-tls" not found
Jul 13 23:39:55.339245 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.339226    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 23:39:55.339245 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.339235    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bddfd4ff-wdnd5: secret "image-registry-tls" not found
Jul 13 23:39:55.339351 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.339272    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls podName:1e8f7563-0a5d-4af9-a105-c1ee533ec1ea nodeName:}" failed. No retries permitted until 2026-07-13 23:39:59.339259009 +0000 UTC m=+47.626477334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls") pod "image-registry-64bddfd4ff-wdnd5" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea") : secret "image-registry-tls" not found
Jul 13 23:39:55.540579 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.540542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:55.540754 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.540605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:55.540754 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.540703    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:55.540854 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.540775    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:57.540755513 +0000 UTC m=+45.827973841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:55.540854 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.540713    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 23:39:55.540854 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:55.540846    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:39:57.540828365 +0000 UTC m=+45.828046688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : configmap "kube-rbac-proxy" not found
Jul 13 23:39:55.685458 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.685369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kpnjw" event={"ID":"99ef6df1-688a-4170-ad90-4cdfdab2528c","Type":"ContainerStarted","Data":"bd502382b615ebe26ece08ad41412215d95864f52c627fcb97bf03ff60ac620b"}
Jul 13 23:39:55.686569 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.686541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq" event={"ID":"7eb9e210-8467-458b-b364-437157f57a02","Type":"ContainerStarted","Data":"99d791a41432ec20f87bb3afd822ee630fc2c195b11b6ec24024c4af6ba92e71"}
Jul 13 23:39:55.700179 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:55.700128    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kpnjw" podStartSLOduration=2.043541082 podStartE2EDuration="4.700110413s" totalImagesPullingTime="2.656569331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:55.699106941 +0000 UTC m=+43.986325286" watchObservedRunningTime="2026-07-13 23:39:55.700110413 +0000 UTC m=+43.987328739"
Jul 13 23:39:56.233550 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.233528    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fbr8"]
Jul 13 23:39:56.235298 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.235280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.237692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.237669    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 13 23:39:56.238605 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.238580    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 13 23:39:56.238692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.238621    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b2p8j\""
Jul 13 23:39:56.238692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.238675    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 13 23:39:56.238692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.238684    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 13 23:39:56.242963 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.242938    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fbr8"]
Jul 13 23:39:56.328635 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.328614    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jzqgh_a434d4f3-48e1-41f7-b82e-5481913fe081/node-ca/0.log"
Jul 13 23:39:56.347948 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.347922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82zf\" (UniqueName: \"kubernetes.io/projected/1ded996d-f11b-4bc3-ac90-0481e578a666-kube-api-access-m82zf\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.348038 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.347964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-key\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.348038 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.348020    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.448422 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.448330    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82zf\" (UniqueName: \"kubernetes.io/projected/1ded996d-f11b-4bc3-ac90-0481e578a666-kube-api-access-m82zf\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.448422 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.448391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-key\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.448594 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.448435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.449068 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.449041    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-cabundle\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.451684 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.451666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ded996d-f11b-4bc3-ac90-0481e578a666-signing-key\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.456847 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.456825    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82zf\" (UniqueName: \"kubernetes.io/projected/1ded996d-f11b-4bc3-ac90-0481e578a666-kube-api-access-m82zf\") pod \"service-ca-79f68b9f4f-4fbr8\" (UID: \"1ded996d-f11b-4bc3-ac90-0481e578a666\") " pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.544131 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.544104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8"
Jul 13 23:39:56.681825 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.681789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-4fbr8"]
Jul 13 23:39:56.685752 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:39:56.685601    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ded996d_f11b_4bc3_ac90_0481e578a666.slice/crio-02f3b05a6838040e812d7bd4707fc2710d5f3a3ba628329a0ec36ec4295756fc WatchSource:0}: Error finding container 02f3b05a6838040e812d7bd4707fc2710d5f3a3ba628329a0ec36ec4295756fc: Status 404 returned error can't find the container with id 02f3b05a6838040e812d7bd4707fc2710d5f3a3ba628329a0ec36ec4295756fc
Jul 13 23:39:56.689623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.689586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8" event={"ID":"1ded996d-f11b-4bc3-ac90-0481e578a666","Type":"ContainerStarted","Data":"02f3b05a6838040e812d7bd4707fc2710d5f3a3ba628329a0ec36ec4295756fc"}
Jul 13 23:39:56.691247 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.691224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq" event={"ID":"7eb9e210-8467-458b-b364-437157f57a02","Type":"ContainerStarted","Data":"6a1ceae47c7c24f013bdd4d0d142490943f79c9162cd94defe612bc5935ed8a2"}
Jul 13 23:39:56.691340 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.691266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq" event={"ID":"7eb9e210-8467-458b-b364-437157f57a02","Type":"ContainerStarted","Data":"dde71a22872f8e86fd274e4121fb1fe839373cf110748d93794a38898e158b81"}
Jul 13 23:39:56.929225 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:56.929190    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kpnjw_99ef6df1-688a-4170-ad90-4cdfdab2528c/serve-healthcheck-canary/0.log"
Jul 13 23:39:57.558419 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:57.558382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:57.559173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:57.558452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:57.559173 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:57.558531    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:57.559173 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:57.558597    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:01.558580614 +0000 UTC m=+49.845798941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : secret "insights-runtime-extractor-tls" not found
Jul 13 23:39:57.559173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:57.559083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77753122-f06d-44ff-a1d4-dc65e5524159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:39:59.371539 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:59.371510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:59.371550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:59.371669    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:59.371680    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:59.371702    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64bddfd4ff-wdnd5: secret "image-registry-tls" not found
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:59.371717    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls podName:3e73bef9-456c-4491-b77f-7b4e95f0b355 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:07.371704048 +0000 UTC m=+55.658922372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls") pod "dns-default-sxkvh" (UID: "3e73bef9-456c-4491-b77f-7b4e95f0b355") : secret "dns-default-metrics-tls" not found
Jul 13 23:39:59.371870 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:39:59.371756    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls podName:1e8f7563-0a5d-4af9-a105-c1ee533ec1ea nodeName:}" failed. No retries permitted until 2026-07-13 23:40:07.371736945 +0000 UTC m=+55.658955275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls") pod "image-registry-64bddfd4ff-wdnd5" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea") : secret "image-registry-tls" not found
Jul 13 23:39:59.702768 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:59.702672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8" event={"ID":"1ded996d-f11b-4bc3-ac90-0481e578a666","Type":"ContainerStarted","Data":"950dfb01880973d0d9b9895c5a266c022c10e7c83c789944da2de433b5ffa6ff"}
Jul 13 23:39:59.722797 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:59.722744    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z5wkq" podStartSLOduration=4.423893588 podStartE2EDuration="5.722722062s" totalImagesPullingTime="1.298828474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:56.741155133 +0000 UTC m=+45.028373479" watchObservedRunningTime="2026-07-13 23:39:59.722722062 +0000 UTC m=+48.009940429"
Jul 13 23:39:59.722959 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:39:59.722843    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-4fbr8" podStartSLOduration=1.512177613 podStartE2EDuration="3.72283632s" totalImagesPullingTime="2.210658707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:39:59.721993671 +0000 UTC m=+48.009212021" watchObservedRunningTime="2026-07-13 23:39:59.72283632 +0000 UTC m=+48.010054667"
Jul 13 23:40:01.588650 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:01.588607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:40:01.589178 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:40:01.588748    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 23:40:01.589178 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:40:01.588811    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls podName:77753122-f06d-44ff-a1d4-dc65e5524159 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:09.588793381 +0000 UTC m=+57.876011764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7ds97" (UID: "77753122-f06d-44ff-a1d4-dc65e5524159") : secret "insights-runtime-extractor-tls" not found
Jul 13 23:40:07.431390 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.431358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:07.431390 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.431394    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:40:07.434532 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.434512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e73bef9-456c-4491-b77f-7b4e95f0b355-metrics-tls\") pod \"dns-default-sxkvh\" (UID: \"3e73bef9-456c-4491-b77f-7b4e95f0b355\") " pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:40:07.434602 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.434534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"image-registry-64bddfd4ff-wdnd5\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") " pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:07.472441 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.472415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:07.500576 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.500549    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:40:07.634712 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.634686    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:40:07.637618 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:07.637591    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8f7563_0a5d_4af9_a105_c1ee533ec1ea.slice/crio-d5e7f74aab112c3b7fa9665437d6e5a59ad9a39f55fd5ca8fa7b0f9eafa5bcb9 WatchSource:0}: Error finding container d5e7f74aab112c3b7fa9665437d6e5a59ad9a39f55fd5ca8fa7b0f9eafa5bcb9: Status 404 returned error can't find the container with id d5e7f74aab112c3b7fa9665437d6e5a59ad9a39f55fd5ca8fa7b0f9eafa5bcb9
Jul 13 23:40:07.646165 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.646144    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxkvh"]
Jul 13 23:40:07.648547 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:07.648527    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e73bef9_456c_4491_b77f_7b4e95f0b355.slice/crio-d5908aaabbe8747a690c51350b0035cda66c3ec85977827df601b0d89fbad580 WatchSource:0}: Error finding container d5908aaabbe8747a690c51350b0035cda66c3ec85977827df601b0d89fbad580: Status 404 returned error can't find the container with id d5908aaabbe8747a690c51350b0035cda66c3ec85977827df601b0d89fbad580
Jul 13 23:40:07.723447 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.723415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxkvh" event={"ID":"3e73bef9-456c-4491-b77f-7b4e95f0b355","Type":"ContainerStarted","Data":"d5908aaabbe8747a690c51350b0035cda66c3ec85977827df601b0d89fbad580"}
Jul 13 23:40:07.724498 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:07.724474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" event={"ID":"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea","Type":"ContainerStarted","Data":"d5e7f74aab112c3b7fa9665437d6e5a59ad9a39f55fd5ca8fa7b0f9eafa5bcb9"}
Jul 13 23:40:08.729329 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:08.729296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" event={"ID":"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea","Type":"ContainerStarted","Data":"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"}
Jul 13 23:40:08.729756 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:08.729432    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:09.652336 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:09.652309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:40:09.654474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:09.654452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77753122-f06d-44ff-a1d4-dc65e5524159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7ds97\" (UID: \"77753122-f06d-44ff-a1d4-dc65e5524159\") " pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:40:09.681462 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:09.681441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7ds97"
Jul 13 23:40:10.018487 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.018438    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" podStartSLOduration=36.018421806 podStartE2EDuration="36.018421806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:08.751783389 +0000 UTC m=+57.039001737" watchObservedRunningTime="2026-07-13 23:40:10.018421806 +0000 UTC m=+58.305640130"
Jul 13 23:40:10.019383 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.019345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7ds97"]
Jul 13 23:40:10.021451 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:10.021427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77753122_f06d_44ff_a1d4_dc65e5524159.slice/crio-6771294dc0919d0e26e5d2956d10855ea8d5ca1c4a9cfaa807c7e1f622f03af8 WatchSource:0}: Error finding container 6771294dc0919d0e26e5d2956d10855ea8d5ca1c4a9cfaa807c7e1f622f03af8: Status 404 returned error can't find the container with id 6771294dc0919d0e26e5d2956d10855ea8d5ca1c4a9cfaa807c7e1f622f03af8
Jul 13 23:40:10.738205 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.738169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ds97" event={"ID":"77753122-f06d-44ff-a1d4-dc65e5524159","Type":"ContainerStarted","Data":"6771294dc0919d0e26e5d2956d10855ea8d5ca1c4a9cfaa807c7e1f622f03af8"}
Jul 13 23:40:10.740622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.740557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxkvh" event={"ID":"3e73bef9-456c-4491-b77f-7b4e95f0b355","Type":"ContainerStarted","Data":"b7a7a973232192efc3b4a2b6f5db1933127a827a0ed692ce97ca944fadc1dfe0"}
Jul 13 23:40:10.740622 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.740606    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxkvh" event={"ID":"3e73bef9-456c-4491-b77f-7b4e95f0b355","Type":"ContainerStarted","Data":"b2125552bb48a3e720ccd1f3a2908c1e721aea241b122b7cdaef4e77470de767"}
Jul 13 23:40:10.741403 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.741384    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:40:10.759212 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:10.759161    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sxkvh" podStartSLOduration=17.506522728 podStartE2EDuration="19.759141714s" totalImagesPullingTime="2.252618986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:10.757880357 +0000 UTC m=+59.045098703" watchObservedRunningTime="2026-07-13 23:40:10.759141714 +0000 UTC m=+59.046360063"
Jul 13 23:40:14.754155 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:14.754081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ds97" event={"ID":"77753122-f06d-44ff-a1d4-dc65e5524159","Type":"ContainerStarted","Data":"1cb45918b851f3fd0e3df6952cf2dce5053ad397d70df227d585fa641e6a9b3f"}
Jul 13 23:40:15.759805 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.759778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ds97" event={"ID":"77753122-f06d-44ff-a1d4-dc65e5524159","Type":"ContainerStarted","Data":"e47f46dd149ed47155fc91d75511922cd3f3be0beee7e8a44a18a9927ba84c12"}
Jul 13 23:40:15.920430 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.920321    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"]
Jul 13 23:40:15.942976 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.942950    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"]
Jul 13 23:40:15.943102 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.943061    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:15.945606 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.945584    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 23:40:15.945730 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.945617    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 23:40:15.945730 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.945700    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 23:40:15.945872 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.945856    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 23:40:15.967156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.967090    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"]
Jul 13 23:40:15.967156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.967153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:15.969787 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.969769    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sfvcf\""
Jul 13 23:40:15.969895 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.969769    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 13 23:40:15.983354 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.983334    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:15.983354 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.983348    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:40:15.983486 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.983389    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"]
Jul 13 23:40:15.983486 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.983400    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"]
Jul 13 23:40:15.983486 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.983414    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"]
Jul 13 23:40:15.985925 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.985909    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 23:40:15.986138 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.986112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 23:40:15.986195 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.986141    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 23:40:15.986195 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:15.986151    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 23:40:16.003713 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.003691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rnm\" (UniqueName: \"kubernetes.io/projected/4ab917a7-0566-40da-a28e-c14508e0a2ef-kube-api-access-h7rnm\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.003824 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.003731    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4ab917a7-0566-40da-a28e-c14508e0a2ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.003824 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.003763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab917a7-0566-40da-a28e-c14508e0a2ef-tmp\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.003923 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.003849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2cr\" (UniqueName: \"kubernetes.io/projected/85163efd-6a91-4cf2-b46a-979414b309f9-kube-api-access-8p2cr\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.003923 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.003881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85163efd-6a91-4cf2-b46a-979414b309f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.021606 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.021586    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c89c8cd49-9fvx8"]
Jul 13 23:40:16.043248 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.043228    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.048252 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.048232    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c89c8cd49-9fvx8"]
Jul 13 23:40:16.105118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-installation-pull-secrets\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdfq\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-kube-api-access-bvdfq\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85163efd-6a91-4cf2-b46a-979414b309f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.105220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff7262e-2b87-4582-a6c8-c66b9f43974b-ca-trust-extracted\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rnm\" (UniqueName: \"kubernetes.io/projected/4ab917a7-0566-40da-a28e-c14508e0a2ef-kube-api-access-h7rnm\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.105369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105301    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-trusted-ca\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4ab917a7-0566-40da-a28e-c14508e0a2ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.105563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-image-registry-private-configuration\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab917a7-0566-40da-a28e-c14508e0a2ef-tmp\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.105563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-bound-sa-token\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105563 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-tls\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105797 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblc5\" (UniqueName: \"kubernetes.io/projected/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-kube-api-access-tblc5\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105797 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.105797 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-certificates\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.105797 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2cr\" (UniqueName: \"kubernetes.io/projected/85163efd-6a91-4cf2-b46a-979414b309f9-kube-api-access-8p2cr\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.106011 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.105816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ab917a7-0566-40da-a28e-c14508e0a2ef-tmp\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.107704 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.107685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/85163efd-6a91-4cf2-b46a-979414b309f9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.108195 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.108179    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4ab917a7-0566-40da-a28e-c14508e0a2ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.118911 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.118872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2cr\" (UniqueName: \"kubernetes.io/projected/85163efd-6a91-4cf2-b46a-979414b309f9-kube-api-access-8p2cr\") pod \"managed-serviceaccount-addon-agent-577979fbcb-sh5dl\" (UID: \"85163efd-6a91-4cf2-b46a-979414b309f9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.119267 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.119247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rnm\" (UniqueName: \"kubernetes.io/projected/4ab917a7-0566-40da-a28e-c14508e0a2ef-kube-api-access-h7rnm\") pod \"klusterlet-addon-workmgr-5b6d85959c-ss8dx\" (UID: \"4ab917a7-0566-40da-a28e-c14508e0a2ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.206705 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.206619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblc5\" (UniqueName: \"kubernetes.io/projected/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-kube-api-access-tblc5\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.206705 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.206670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.206937 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.206806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-certificates\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.206998 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.206965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-installation-pull-secrets\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207050 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.207103 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdfq\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-kube-api-access-bvdfq\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207157 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.207157 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207129    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff7262e-2b87-4582-a6c8-c66b9f43974b-ca-trust-extracted\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207258 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.207258 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-trusted-ca\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207245    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-image-registry-private-configuration\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-bound-sa-token\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207333    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.207549 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-tls\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.207608 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.207734 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.207710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-certificates\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.208513 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.208486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ff7262e-2b87-4582-a6c8-c66b9f43974b-ca-trust-extracted\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.209395 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.209369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ff7262e-2b87-4582-a6c8-c66b9f43974b-trusted-ca\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.210308 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.210282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-ca\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.210679 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.210659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.210779 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.210762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-registry-tls\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.210945 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.210923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-installation-pull-secrets\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.211009 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.210979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.212553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.212533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ff7262e-2b87-4582-a6c8-c66b9f43974b-image-registry-private-configuration\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.212648 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.212563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-hub\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.215675 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.215653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdfq\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-kube-api-access-bvdfq\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.215860 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.215833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblc5\" (UniqueName: \"kubernetes.io/projected/ff9e03a9-fb0c-47d5-815b-1fa7778e5043-kube-api-access-tblc5\") pod \"cluster-proxy-proxy-agent-9b7b6847f-rwbq6\" (UID: \"ff9e03a9-fb0c-47d5-815b-1fa7778e5043\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.216365 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.216337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ff7262e-2b87-4582-a6c8-c66b9f43974b-bound-sa-token\") pod \"image-registry-7c89c8cd49-9fvx8\" (UID: \"9ff7262e-2b87-4582-a6c8-c66b9f43974b\") " pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.251559 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.251528    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:16.293587 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.293562    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"
Jul 13 23:40:16.300779 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.300723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"
Jul 13 23:40:16.367019 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.366990    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.435921 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.435811    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"]
Jul 13 23:40:16.438776 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:16.438745    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab917a7_0566_40da_a28e_c14508e0a2ef.slice/crio-03d277305351d9dda1aadcfd716584642f9b73de55d85577d446e45f0e3a045b WatchSource:0}: Error finding container 03d277305351d9dda1aadcfd716584642f9b73de55d85577d446e45f0e3a045b: Status 404 returned error can't find the container with id 03d277305351d9dda1aadcfd716584642f9b73de55d85577d446e45f0e3a045b
Jul 13 23:40:16.482400 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.482370    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl"]
Jul 13 23:40:16.488067 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:16.488035    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85163efd_6a91_4cf2_b46a_979414b309f9.slice/crio-5ef5ca1a4bd09caf67ca01e309b82d08a4e0729e52adaab9210ca60faa4a4c25 WatchSource:0}: Error finding container 5ef5ca1a4bd09caf67ca01e309b82d08a4e0729e52adaab9210ca60faa4a4c25: Status 404 returned error can't find the container with id 5ef5ca1a4bd09caf67ca01e309b82d08a4e0729e52adaab9210ca60faa4a4c25
Jul 13 23:40:16.517739 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.517712    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6"]
Jul 13 23:40:16.526277 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:16.526249    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff9e03a9_fb0c_47d5_815b_1fa7778e5043.slice/crio-0ecbffd47b51a840461d25d538f49dd7f423a6a826d2a1305e9f684c44cbd791 WatchSource:0}: Error finding container 0ecbffd47b51a840461d25d538f49dd7f423a6a826d2a1305e9f684c44cbd791: Status 404 returned error can't find the container with id 0ecbffd47b51a840461d25d538f49dd7f423a6a826d2a1305e9f684c44cbd791
Jul 13 23:40:16.573301 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.573275    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c89c8cd49-9fvx8"]
Jul 13 23:40:16.576831 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:16.576805    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff7262e_2b87_4582_a6c8_c66b9f43974b.slice/crio-f1ac2a40062c491fb26f0a2a5027feb4c56c71d810b03a2395ffb965e6371b2d WatchSource:0}: Error finding container f1ac2a40062c491fb26f0a2a5027feb4c56c71d810b03a2395ffb965e6371b2d: Status 404 returned error can't find the container with id f1ac2a40062c491fb26f0a2a5027feb4c56c71d810b03a2395ffb965e6371b2d
Jul 13 23:40:16.668969 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.668947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwxsn"
Jul 13 23:40:16.765144 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.765114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl" event={"ID":"85163efd-6a91-4cf2-b46a-979414b309f9","Type":"ContainerStarted","Data":"5ef5ca1a4bd09caf67ca01e309b82d08a4e0729e52adaab9210ca60faa4a4c25"}
Jul 13 23:40:16.766519 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.766488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6" event={"ID":"ff9e03a9-fb0c-47d5-815b-1fa7778e5043","Type":"ContainerStarted","Data":"0ecbffd47b51a840461d25d538f49dd7f423a6a826d2a1305e9f684c44cbd791"}
Jul 13 23:40:16.768009 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.767988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx" event={"ID":"4ab917a7-0566-40da-a28e-c14508e0a2ef","Type":"ContainerStarted","Data":"03d277305351d9dda1aadcfd716584642f9b73de55d85577d446e45f0e3a045b"}
Jul 13 23:40:16.769755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.769731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8" event={"ID":"9ff7262e-2b87-4582-a6c8-c66b9f43974b","Type":"ContainerStarted","Data":"57491c3ba462d804bdba8749d10c135cf58038319b5c7351afe5e9946863fb1f"}
Jul 13 23:40:16.769845 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.769772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8" event={"ID":"9ff7262e-2b87-4582-a6c8-c66b9f43974b","Type":"ContainerStarted","Data":"f1ac2a40062c491fb26f0a2a5027feb4c56c71d810b03a2395ffb965e6371b2d"}
Jul 13 23:40:16.769914 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.769898    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:16.790051 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:16.789977    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8" podStartSLOduration=1.7899601029999999 podStartE2EDuration="1.789960103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:16.788939337 +0000 UTC m=+65.076157685" watchObservedRunningTime="2026-07-13 23:40:16.789960103 +0000 UTC m=+65.077178442"
Jul 13 23:40:17.417142 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.416999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:40:17.422419 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.422131    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 23:40:17.435340 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.434975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c1f071-28f4-4448-9c76-bddfc89bf893-metrics-certs\") pod \"network-metrics-daemon-pqr8z\" (UID: \"26c1f071-28f4-4448-9c76-bddfc89bf893\") " pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:40:17.444744 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.444576    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5s29d\""
Jul 13 23:40:17.453333 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.453108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pqr8z"
Jul 13 23:40:17.518115 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.517656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:40:17.520422 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.520225    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 23:40:17.537348 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.537295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/caccffa7-8996-48b5-8eb1-1e83de888826-original-pull-secret\") pod \"global-pull-secret-syncer-9fw4w\" (UID: \"caccffa7-8996-48b5-8eb1-1e83de888826\") " pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:40:17.619938 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.619904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:40:17.622965 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.622751    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 23:40:17.633773 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.633577    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 23:40:17.645406 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.645348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvt42\" (UniqueName: \"kubernetes.io/projected/836a5748-117e-42ca-9b40-aed9fbbb4807-kube-api-access-qvt42\") pod \"network-check-target-cjj8j\" (UID: \"836a5748-117e-42ca-9b40-aed9fbbb4807\") " pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:40:17.678221 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.678099    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pqr8z"]
Jul 13 23:40:17.752306 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.752075    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qp66b\""
Jul 13 23:40:17.754540 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.754327    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fw4w"
Jul 13 23:40:17.759927 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:17.759850    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:40:20.322556 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:20.322525    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c1f071_28f4_4448_9c76_bddfc89bf893.slice/crio-39112f0dd0e54ff6cb50603395a0eae0a2c59dcd2c69ca10c582b0b6a77e99e1 WatchSource:0}: Error finding container 39112f0dd0e54ff6cb50603395a0eae0a2c59dcd2c69ca10c582b0b6a77e99e1: Status 404 returned error can't find the container with id 39112f0dd0e54ff6cb50603395a0eae0a2c59dcd2c69ca10c582b0b6a77e99e1
Jul 13 23:40:20.791306 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:20.791267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pqr8z" event={"ID":"26c1f071-28f4-4448-9c76-bddfc89bf893","Type":"ContainerStarted","Data":"39112f0dd0e54ff6cb50603395a0eae0a2c59dcd2c69ca10c582b0b6a77e99e1"}
Jul 13 23:40:21.749623 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:21.749595    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sxkvh"
Jul 13 23:40:23.088786 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.088758    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fw4w"]
Jul 13 23:40:23.108063 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.107988    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjj8j"]
Jul 13 23:40:23.201237 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:23.201019    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836a5748_117e_42ca_9b40_aed9fbbb4807.slice/crio-ac1ab08a0d7f7eab60abb1ad92eae4e0315f813c0242d7d7f5ffd78b6e0f4562 WatchSource:0}: Error finding container ac1ab08a0d7f7eab60abb1ad92eae4e0315f813c0242d7d7f5ffd78b6e0f4562: Status 404 returned error can't find the container with id ac1ab08a0d7f7eab60abb1ad92eae4e0315f813c0242d7d7f5ffd78b6e0f4562
Jul 13 23:40:23.807195 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.807154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjj8j" event={"ID":"836a5748-117e-42ca-9b40-aed9fbbb4807","Type":"ContainerStarted","Data":"ac1ab08a0d7f7eab60abb1ad92eae4e0315f813c0242d7d7f5ffd78b6e0f4562"}
Jul 13 23:40:23.810577 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.810498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7ds97" event={"ID":"77753122-f06d-44ff-a1d4-dc65e5524159","Type":"ContainerStarted","Data":"8bb9aa791f8471d7a6bc1906d2261333c141b4cf90e9387c2af24463232f2e35"}
Jul 13 23:40:23.813230 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.812610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx" event={"ID":"4ab917a7-0566-40da-a28e-c14508e0a2ef","Type":"ContainerStarted","Data":"6eb4a813b8cfbff1a7031b7ec0550ca38004cf5b58e1c11b7b49c38d73ff1fe7"}
Jul 13 23:40:23.813230 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.813066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:23.814693 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.814649    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx"
Jul 13 23:40:23.816691 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.816274    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pqr8z" event={"ID":"26c1f071-28f4-4448-9c76-bddfc89bf893","Type":"ContainerStarted","Data":"63f165097e2dc5497fb19ba7c429e68222f1919af266b3795e9aa33efeb2daf3"}
Jul 13 23:40:23.816691 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.816317    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pqr8z" event={"ID":"26c1f071-28f4-4448-9c76-bddfc89bf893","Type":"ContainerStarted","Data":"c6cce6f6fc855ddf441bd858def9cbbbd75c8e39c22992c87886128902727f54"}
Jul 13 23:40:23.817980 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.817926    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fw4w" event={"ID":"caccffa7-8996-48b5-8eb1-1e83de888826","Type":"ContainerStarted","Data":"524ede17b481a6818f5d8ba893f25075d512a63e4d1ac7ca54cad6b38b72b05a"}
Jul 13 23:40:23.819945 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.819919    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl" event={"ID":"85163efd-6a91-4cf2-b46a-979414b309f9","Type":"ContainerStarted","Data":"33e2c5c56f1c64b2bd50c2e54b602a49dc38e6accfba60501ec67f3064bc7404"}
Jul 13 23:40:23.823734 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.823621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6" event={"ID":"ff9e03a9-fb0c-47d5-815b-1fa7778e5043","Type":"ContainerStarted","Data":"a3622f47bea37486d298f382a6c6bd474e976efd2ac054581e84820de1fa5faa"}
Jul 13 23:40:23.823734 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.823661    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6" event={"ID":"ff9e03a9-fb0c-47d5-815b-1fa7778e5043","Type":"ContainerStarted","Data":"a7ec3dd3126f3603851918d60af5574176d5fab82d917f4b64b7d646baf9dbbd"}
Jul 13 23:40:23.823734 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.823682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6" event={"ID":"ff9e03a9-fb0c-47d5-815b-1fa7778e5043","Type":"ContainerStarted","Data":"300c8d67f8bde27e468c335f6e46cc6d31cbcc4f2bb51d8d1aa139f51333fdf4"}
Jul 13 23:40:23.851900 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.851833    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7ds97" podStartSLOduration=18.14726082 podStartE2EDuration="30.851813802s" totalImagesPullingTime="12.704552982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:23.850195429 +0000 UTC m=+72.137413776" watchObservedRunningTime="2026-07-13 23:40:23.851813802 +0000 UTC m=+72.139032150"
Jul 13 23:40:23.882795 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.882757    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-577979fbcb-sh5dl" podStartSLOduration=2.171462398 podStartE2EDuration="8.882745052s" totalImagesPullingTime="6.711282654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:23.866396218 +0000 UTC m=+72.153614564" watchObservedRunningTime="2026-07-13 23:40:23.882745052 +0000 UTC m=+72.169963398"
Jul 13 23:40:23.883596 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.883562    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b6d85959c-ss8dx" podStartSLOduration=2.121062981 podStartE2EDuration="8.883551132s" totalImagesPullingTime="6.762488151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:23.882417249 +0000 UTC m=+72.169635598" watchObservedRunningTime="2026-07-13 23:40:23.883551132 +0000 UTC m=+72.170769475"
Jul 13 23:40:23.901269 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.901219    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b7b6847f-rwbq6" podStartSLOduration=2.225270592 podStartE2EDuration="8.901201461s" totalImagesPullingTime="6.675930869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:23.900119113 +0000 UTC m=+72.187337460" watchObservedRunningTime="2026-07-13 23:40:23.901201461 +0000 UTC m=+72.188419807"
Jul 13 23:40:23.920484 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:23.920443    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pqr8z" podStartSLOduration=67.01369089 podStartE2EDuration="1m9.920431036s" totalImagesPullingTime="2.906740146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:23.91913423 +0000 UTC m=+72.206352577" watchObservedRunningTime="2026-07-13 23:40:23.920431036 +0000 UTC m=+72.207649382"
Jul 13 23:40:25.992266 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:25.992226    2558 patch_prober.go:28] interesting pod/image-registry-64bddfd4ff-wdnd5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 13 23:40:25.992721 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:25.992289    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 23:40:27.841332 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:27.841297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjj8j" event={"ID":"836a5748-117e-42ca-9b40-aed9fbbb4807","Type":"ContainerStarted","Data":"49e6fc077b8cb760a2046180498d216b11e491e24c9a53e6a4b59630c19b0842"}
Jul 13 23:40:27.841787 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:27.841357    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:40:27.842660 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:27.842638    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fw4w" event={"ID":"caccffa7-8996-48b5-8eb1-1e83de888826","Type":"ContainerStarted","Data":"c43860b1167a0eb111f47a296e678b8b32eb7aa62dea953889168da5b9e89261"}
Jul 13 23:40:27.856539 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:27.856499    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cjj8j" podStartSLOduration=69.584905723 podStartE2EDuration="1m13.856486632s" totalImagesPullingTime="4.271580909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:27.856027686 +0000 UTC m=+76.143246035" watchObservedRunningTime="2026-07-13 23:40:27.856486632 +0000 UTC m=+76.143705012"
Jul 13 23:40:27.869774 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:27.869735    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9fw4w" podStartSLOduration=69.58785917 podStartE2EDuration="1m13.869723171s" totalImagesPullingTime="4.281864001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 23:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:27.869199337 +0000 UTC m=+76.156417674" watchObservedRunningTime="2026-07-13 23:40:27.869723171 +0000 UTC m=+76.156941517"
Jul 13 23:40:31.556845 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.556802    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hjw8c"]
Jul 13 23:40:31.560360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.560336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.564114 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.564088    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 23:40:31.564229 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.564176    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 23:40:31.565675 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.565653    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 23:40:31.565782 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.565721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 23:40:31.565869 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.565854    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7b4pc\""
Jul 13 23:40:31.566009 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.565987    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 23:40:31.566272 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.566252    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 23:40:31.632018 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.631976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-root\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632156 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-metrics-client-ca\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-textfile\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbchj\" (UniqueName: \"kubernetes.io/projected/9263031a-45fa-44af-8f76-33d260f4dc08-kube-api-access-zbchj\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632356 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-wtmp\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.632552 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.632376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-sys\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.732973 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.732945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-sys\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.732980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-root\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733047    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-sys\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733104 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733071    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-metrics-client-ca\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-root\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-textfile\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbchj\" (UniqueName: \"kubernetes.io/projected/9263031a-45fa-44af-8f76-33d260f4dc08-kube-api-access-zbchj\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-wtmp\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-wtmp\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733650 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.733391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-textfile\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.733650 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:40:31.733431    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 23:40:31.733650 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:40:31.733484    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls podName:9263031a-45fa-44af-8f76-33d260f4dc08 nodeName:}" failed. No retries permitted until 2026-07-13 23:40:32.233465036 +0000 UTC m=+80.520683363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls") pod "node-exporter-hjw8c" (UID: "9263031a-45fa-44af-8f76-33d260f4dc08") : secret "node-exporter-tls" not found
Jul 13 23:40:31.734227 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.734200    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.734457 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.734440    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9263031a-45fa-44af-8f76-33d260f4dc08-metrics-client-ca\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.735848 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.735828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:31.747677 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:31.747654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbchj\" (UniqueName: \"kubernetes.io/projected/9263031a-45fa-44af-8f76-33d260f4dc08-kube-api-access-zbchj\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:32.238085 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:32.238053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:32.240299 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:32.240273    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9263031a-45fa-44af-8f76-33d260f4dc08-node-exporter-tls\") pod \"node-exporter-hjw8c\" (UID: \"9263031a-45fa-44af-8f76-33d260f4dc08\") " pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:32.473424 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:32.473393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjw8c"
Jul 13 23:40:32.485331 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:40:32.485303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9263031a_45fa_44af_8f76_33d260f4dc08.slice/crio-74ed64d6c681e68bd415cd9ff514e73add417bcc325959992591865f1957b895 WatchSource:0}: Error finding container 74ed64d6c681e68bd415cd9ff514e73add417bcc325959992591865f1957b895: Status 404 returned error can't find the container with id 74ed64d6c681e68bd415cd9ff514e73add417bcc325959992591865f1957b895
Jul 13 23:40:32.860780 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:32.860737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjw8c" event={"ID":"9263031a-45fa-44af-8f76-33d260f4dc08","Type":"ContainerStarted","Data":"74ed64d6c681e68bd415cd9ff514e73add417bcc325959992591865f1957b895"}
Jul 13 23:40:33.865784 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:33.865749    2558 generic.go:356] "Generic (PLEG): container finished" podID="9263031a-45fa-44af-8f76-33d260f4dc08" containerID="7811490f72e5093e06bc28e9372096b5d75b7dd9d5b6517421c14524b469f09e" exitCode=0
Jul 13 23:40:33.866167 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:33.865831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjw8c" event={"ID":"9263031a-45fa-44af-8f76-33d260f4dc08","Type":"ContainerDied","Data":"7811490f72e5093e06bc28e9372096b5d75b7dd9d5b6517421c14524b469f09e"}
Jul 13 23:40:34.871241 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:34.871207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjw8c" event={"ID":"9263031a-45fa-44af-8f76-33d260f4dc08","Type":"ContainerStarted","Data":"0f70218e5b70c666ad5f8af21c2a31b140d3eb5d3f44bcc76a6f20dd72dd53b3"}
Jul 13 23:40:34.871651 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:34.871258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjw8c" event={"ID":"9263031a-45fa-44af-8f76-33d260f4dc08","Type":"ContainerStarted","Data":"c2564df0db06a601fc99801a34e827b288b68cb2507f03c104cacfd69b77784d"}
Jul 13 23:40:34.889733 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:34.889683    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hjw8c" podStartSLOduration=2.995486915 podStartE2EDuration="3.889666069s" totalImagesPullingTime="831.425192ms" totalInitContainerRuntime="62.753962ms" isStatefulPod=true podCreationTimestamp="2026-07-13 23:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:40:34.88856428 +0000 UTC m=+83.175782627" watchObservedRunningTime="2026-07-13 23:40:34.889666069 +0000 UTC m=+83.176884425"
Jul 13 23:40:35.988340 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:35.988315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:37.779927 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:37.779882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c89c8cd49-9fvx8"
Jul 13 23:40:41.003881 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.003843    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerName="registry" containerID="cri-o://03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685" gracePeriod=30
Jul 13 23:40:41.240469 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.240448    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:41.305072 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brzxr\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305120    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305383 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305342    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305435 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305417    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305494 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305461    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration\") pod \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\" (UID: \"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea\") "
Jul 13 23:40:41.305545 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305516    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 23:40:41.305716 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.305685    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-certificates\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.306046 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.306017    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 23:40:41.307736 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.307691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:40:41.307847 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.307699    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:40:41.307847 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.307790    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr" (OuterVolumeSpecName: "kube-api-access-brzxr") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "kube-api-access-brzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:40:41.307847 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.307832    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:40:41.308011 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.307972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:40:41.314024 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.313999    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" (UID: "1e8f7563-0a5d-4af9-a105-c1ee533ec1ea"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:40:41.406550 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406508    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brzxr\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-kube-api-access-brzxr\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406550 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406549    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-bound-sa-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406723 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406562    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-registry-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406723 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406574    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-installation-pull-secrets\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406723 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406586    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-ca-trust-extracted\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406723 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406598    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-trusted-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.406723 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.406611    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea-image-registry-private-configuration\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:40:41.894963 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.894929    2558 generic.go:356] "Generic (PLEG): container finished" podID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerID="03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685" exitCode=0
Jul 13 23:40:41.895118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.894971    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5"
Jul 13 23:40:41.895118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.894995    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" event={"ID":"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea","Type":"ContainerDied","Data":"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"}
Jul 13 23:40:41.895118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.895034    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64bddfd4ff-wdnd5" event={"ID":"1e8f7563-0a5d-4af9-a105-c1ee533ec1ea","Type":"ContainerDied","Data":"d5e7f74aab112c3b7fa9665437d6e5a59ad9a39f55fd5ca8fa7b0f9eafa5bcb9"}
Jul 13 23:40:41.895118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.895053    2558 scope.go:117] "RemoveContainer" containerID="03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"
Jul 13 23:40:41.906619 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.906597    2558 scope.go:117] "RemoveContainer" containerID="03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"
Jul 13 23:40:41.906959 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:40:41.906928    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685\": container with ID starting with 03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685 not found: ID does not exist" containerID="03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"
Jul 13 23:40:41.907044 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.906958    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685"} err="failed to get container status \"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685\": rpc error: code = NotFound desc = could not find container \"03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685\": container with ID starting with 03eb8c0f75ae3fd6f979a80e9cf290a08edf1f971723f4c6e68f3a7518683685 not found: ID does not exist"
Jul 13 23:40:41.912352 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.912334    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:40:41.915861 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:41.915840    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64bddfd4ff-wdnd5"]
Jul 13 23:40:43.429700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:43.429671    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" path="/var/lib/kubelet/pods/1e8f7563-0a5d-4af9-a105-c1ee533ec1ea/volumes"
Jul 13 23:40:58.849075 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:40:58.849043    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cjj8j"
Jul 13 23:41:04.066935 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:41:04.066881    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kpnjw_99ef6df1-688a-4170-ad90-4cdfdab2528c/serve-healthcheck-canary/0.log"
Jul 13 23:43:53.061065 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.060964    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"]
Jul 13 23:43:53.061773 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.061437    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerName="registry"
Jul 13 23:43:53.061773 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.061453    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerName="registry"
Jul 13 23:43:53.061773 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.061569    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8f7563-0a5d-4af9-a105-c1ee533ec1ea" containerName="registry"
Jul 13 23:43:53.064006 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.063977    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.067028 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.067009    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 13 23:43:53.067141 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.067097    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 13 23:43:53.067217 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.067204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 13 23:43:53.067404 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.067390    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 13 23:43:53.067489 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.067473    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x2sjv\""
Jul 13 23:43:53.085252 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.085228    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"]
Jul 13 23:43:53.164052 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.164030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd4z\" (UniqueName: \"kubernetes.io/projected/d501f675-1528-402a-aabf-31a843897b39-kube-api-access-jcd4z\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.164189 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.164070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-tmp\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.164189 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.164108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-manifests\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.164189 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.164126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.164329 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.164205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.264794 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.264770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd4z\" (UniqueName: \"kubernetes.io/projected/d501f675-1528-402a-aabf-31a843897b39-kube-api-access-jcd4z\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.264924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.264804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-tmp\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.264924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.264835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-manifests\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.264924 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.264853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.265090 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.264932    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.265198 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.265180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-manifests\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.267226 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.267204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d501f675-1528-402a-aabf-31a843897b39-tmp\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.267333 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.267262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.267392 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.267340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d501f675-1528-402a-aabf-31a843897b39-webhook-cert\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.274651 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.274633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd4z\" (UniqueName: \"kubernetes.io/projected/d501f675-1528-402a-aabf-31a843897b39-kube-api-access-jcd4z\") pod \"opendatahub-operator-controller-manager-74bf4db56c-7xd8q\" (UID: \"d501f675-1528-402a-aabf-31a843897b39\") " pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.373727 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.373668    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:53.509671 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:53.509638    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"]
Jul 13 23:43:53.512601 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:43:53.512548    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd501f675_1528_402a_aabf_31a843897b39.slice/crio-680194e91294534d879b6485515c3dbf7c6b7d273d9069f097a10f463cb16c98 WatchSource:0}: Error finding container 680194e91294534d879b6485515c3dbf7c6b7d273d9069f097a10f463cb16c98: Status 404 returned error can't find the container with id 680194e91294534d879b6485515c3dbf7c6b7d273d9069f097a10f463cb16c98
Jul 13 23:43:54.516482 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:54.516439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q" event={"ID":"d501f675-1528-402a-aabf-31a843897b39","Type":"ContainerStarted","Data":"680194e91294534d879b6485515c3dbf7c6b7d273d9069f097a10f463cb16c98"}
Jul 13 23:43:56.524425 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:56.524392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q" event={"ID":"d501f675-1528-402a-aabf-31a843897b39","Type":"ContainerStarted","Data":"01c5fc3d49df18c4604cea57f54a7a5a382ce73894a0fe58283facd469857a3f"}
Jul 13 23:43:58.531331 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:58.531300    2558 generic.go:356] "Generic (PLEG): container finished" podID="d501f675-1528-402a-aabf-31a843897b39" containerID="01c5fc3d49df18c4604cea57f54a7a5a382ce73894a0fe58283facd469857a3f" exitCode=0
Jul 13 23:43:58.531670 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:58.531373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q" event={"ID":"d501f675-1528-402a-aabf-31a843897b39","Type":"ContainerDied","Data":"01c5fc3d49df18c4604cea57f54a7a5a382ce73894a0fe58283facd469857a3f"}
Jul 13 23:43:59.536612 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:59.536580    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q" event={"ID":"d501f675-1528-402a-aabf-31a843897b39","Type":"ContainerStarted","Data":"6af2be25dfdd0e834aab4322b3a1599125f3fe4793c7fbcf9e59ce8a07898876"}
Jul 13 23:43:59.537001 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:59.536787    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:43:59.557679 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:43:59.557623    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q" podStartSLOduration=2.297069536 podStartE2EDuration="6.557604626s" totalImagesPullingTime="3.186813629s" totalInitContainerRuntime="1.073721461s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:43:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:43:59.556655839 +0000 UTC m=+287.843874184" watchObservedRunningTime="2026-07-13 23:43:59.557604626 +0000 UTC m=+287.844822990"
Jul 13 23:44:10.541267 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:10.541240    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74bf4db56c-7xd8q"
Jul 13 23:44:12.141191 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.141161    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 23:44:12.970090 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.970057    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lq7qn"]
Jul 13 23:44:12.976658 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.976634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:12.979730 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.979709    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7xdjc\""
Jul 13 23:44:12.979974 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.979961    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 13 23:44:12.994507 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:12.994485    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lq7qn"]
Jul 13 23:44:13.101907 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.101865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl66p\" (UniqueName: \"kubernetes.io/projected/c40e8682-71f9-4799-83b1-b48b0d9de463-kube-api-access-fl66p\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.102070 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.102035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.203326 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.203296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl66p\" (UniqueName: \"kubernetes.io/projected/c40e8682-71f9-4799-83b1-b48b0d9de463-kube-api-access-fl66p\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.203738 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.203419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.203738 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:44:13.203555    2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 13 23:44:13.203738 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:44:13.203608    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert podName:c40e8682-71f9-4799-83b1-b48b0d9de463 nodeName:}" failed. No retries permitted until 2026-07-13 23:44:13.703592393 +0000 UTC m=+301.990810717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert") pod "odh-model-controller-7998c67dc4-lq7qn" (UID: "c40e8682-71f9-4799-83b1-b48b0d9de463") : secret "odh-model-controller-webhook-cert" not found
Jul 13 23:44:13.218466 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.218432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl66p\" (UniqueName: \"kubernetes.io/projected/c40e8682-71f9-4799-83b1-b48b0d9de463-kube-api-access-fl66p\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.706667 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.706639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.708794 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.708723    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c40e8682-71f9-4799-83b1-b48b0d9de463-cert\") pod \"odh-model-controller-7998c67dc4-lq7qn\" (UID: \"c40e8682-71f9-4799-83b1-b48b0d9de463\") " pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:13.886502 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:13.886470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:14.031271 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.031237    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lq7qn"]
Jul 13 23:44:14.033368 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:44:14.033343    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40e8682_71f9_4799_83b1_b48b0d9de463.slice/crio-efa3c330bfc3b21180034e67a5c32341da990e21d0460a4d7ae365cee0d58f58 WatchSource:0}: Error finding container efa3c330bfc3b21180034e67a5c32341da990e21d0460a4d7ae365cee0d58f58: Status 404 returned error can't find the container with id efa3c330bfc3b21180034e67a5c32341da990e21d0460a4d7ae365cee0d58f58
Jul 13 23:44:14.326870 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.326836    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:14.330978 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.330960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.333407 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 13 23:44:14.333533 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 13 23:44:14.333533 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tlcwd\""
Jul 13 23:44:14.333762 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 23:44:14.333762 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 13 23:44:14.333963 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.333762    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 23:44:14.334335 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.334321    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 23:44:14.344092 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.344072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:14.417981 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.417953    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418095 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.417988    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418095 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.418026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418095 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.418088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg57x\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418251 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.418132    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418251 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.418162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.418251 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.418189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519100 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg57x\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519215 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519215 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519215 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519215 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519423 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519423 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.519764 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.519735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.521673 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.521650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.521756 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.521670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.521756 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.521649    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.521835 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.521795    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.527849 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.527830    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.528488 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.528468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg57x\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x\") pod \"istiod-openshift-gateway-7574459656-szgzn\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.585296 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.585239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn" event={"ID":"c40e8682-71f9-4799-83b1-b48b0d9de463","Type":"ContainerStarted","Data":"efa3c330bfc3b21180034e67a5c32341da990e21d0460a4d7ae365cee0d58f58"}
Jul 13 23:44:14.640517 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.640495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:14.779314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.779169    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:14.783256 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:44:14.783224    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd764af95_ecf6_47b7_ba85_86896e32c61f.slice/crio-9110eabfa1d27bad15c6fd948cd1a16a91f35d8c19f746bb2d965fc6c8566a18 WatchSource:0}: Error finding container 9110eabfa1d27bad15c6fd948cd1a16a91f35d8c19f746bb2d965fc6c8566a18: Status 404 returned error can't find the container with id 9110eabfa1d27bad15c6fd948cd1a16a91f35d8c19f746bb2d965fc6c8566a18
Jul 13 23:44:14.785939 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:14.785902    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 23:44:15.590969 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:15.590932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" event={"ID":"d764af95-ecf6-47b7-ba85-86896e32c61f","Type":"ContainerStarted","Data":"9110eabfa1d27bad15c6fd948cd1a16a91f35d8c19f746bb2d965fc6c8566a18"}
Jul 13 23:44:17.600653 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:17.600612    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn" event={"ID":"c40e8682-71f9-4799-83b1-b48b0d9de463","Type":"ContainerStarted","Data":"eca61893920cf90cfe87805eaccb6957bfbc95cd610c4d7f091a63ef9abc2b3f"}
Jul 13 23:44:17.601109 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:17.600707    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:44:17.617494 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:17.617417    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn" podStartSLOduration=2.836806322 podStartE2EDuration="5.617397251s" totalImagesPullingTime="2.780590929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:44:17.615828667 +0000 UTC m=+305.903047014" watchObservedRunningTime="2026-07-13 23:44:17.617397251 +0000 UTC m=+305.904615599"
Jul 13 23:44:18.603290 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:18.603251    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:44:18.603656 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:18.603331    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:44:19.609292 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:19.609259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" event={"ID":"d764af95-ecf6-47b7-ba85-86896e32c61f","Type":"ContainerStarted","Data":"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"}
Jul 13 23:44:19.609695 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:19.609389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:19.610963 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:19.610931    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-szgzn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 23:44:19.611088 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:19.610981    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 23:44:19.635995 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:19.635951    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" podStartSLOduration=1.819009244 podStartE2EDuration="5.635937013s" totalImagesPullingTime="3.816927769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:44:19.633816643 +0000 UTC m=+307.921034990" watchObservedRunningTime="2026-07-13 23:44:19.635937013 +0000 UTC m=+307.923155358"
Jul 13 23:44:20.613568 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:20.613542    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:21.838455 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.838422    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"]
Jul 13 23:44:21.841643 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.841628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.861008 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.860981    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"]
Jul 13 23:44:21.985061 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985216 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4d1aec-49f0-457c-8cce-17dc69d2349c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985216 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985216 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985216 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985183    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985216 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjv2\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-kube-api-access-sfjv2\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:21.985375 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:21.985238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085836 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085993 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085993 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4d1aec-49f0-457c-8cce-17dc69d2349c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085993 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085993 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.085993 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.085977    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.086239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.086022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjv2\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-kube-api-access-sfjv2\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.086696 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.086659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.088281 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.088250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.088393 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.088307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.088507 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.088471    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4d1aec-49f0-457c-8cce-17dc69d2349c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.088687 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.088666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.094934 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.094912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.095020 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.095004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjv2\" (UniqueName: \"kubernetes.io/projected/6b4d1aec-49f0-457c-8cce-17dc69d2349c-kube-api-access-sfjv2\") pod \"istiod-openshift-gateway-94bb8fbfd-k5p7r\" (UID: \"6b4d1aec-49f0-457c-8cce-17dc69d2349c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.149893 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.149872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.294147 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.294121    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"]
Jul 13 23:44:22.296302 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:44:22.296273    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4d1aec_49f0_457c_8cce_17dc69d2349c.slice/crio-31ec559d4661506546c64decc82a6097c34b2361a484876802a89396d1a2b143 WatchSource:0}: Error finding container 31ec559d4661506546c64decc82a6097c34b2361a484876802a89396d1a2b143: Status 404 returned error can't find the container with id 31ec559d4661506546c64decc82a6097c34b2361a484876802a89396d1a2b143
Jul 13 23:44:22.299404 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.299373    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:44:22.299488 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.299429    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:44:22.622634 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.622507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r" event={"ID":"6b4d1aec-49f0-457c-8cce-17dc69d2349c","Type":"ContainerStarted","Data":"def13e8c5293a50c7a3dc127c6052b8750bf4ca4fbbc2c049d0983eb691e9567"}
Jul 13 23:44:22.622634 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.622576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r" event={"ID":"6b4d1aec-49f0-457c-8cce-17dc69d2349c","Type":"ContainerStarted","Data":"31ec559d4661506546c64decc82a6097c34b2361a484876802a89396d1a2b143"}
Jul 13 23:44:22.622851 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.622714    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:22.649761 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:22.649709    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r" podStartSLOduration=1.649691508 podStartE2EDuration="1.649691508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:44:22.64908082 +0000 UTC m=+310.936299177" watchObservedRunningTime="2026-07-13 23:44:22.649691508 +0000 UTC m=+310.936909854"
Jul 13 23:44:23.627134 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:23.627106    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-k5p7r"
Jul 13 23:44:23.711343 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:23.711316    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:23.711517 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:23.711469    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerName="discovery" containerID="cri-o://f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858" gracePeriod=30
Jul 13 23:44:23.954126 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:23.954097    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:24.103304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103273    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103303    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg57x\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103476 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103476 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103361    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103580 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103562    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103633 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103600    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103685 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103634    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig\") pod \"d764af95-ecf6-47b7-ba85-86896e32c61f\" (UID: \"d764af95-ecf6-47b7-ba85-86896e32c61f\") "
Jul 13 23:44:24.103760 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103723    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 23:44:24.103928 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.103909    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-ca-configmap\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.105637 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.105607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:44:24.105749 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.105641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:44:24.105921 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.105878    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts" (OuterVolumeSpecName: "cacerts") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:44:24.105979 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.105947    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs" (OuterVolumeSpecName: "local-certs") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:44:24.106017 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.105985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x" (OuterVolumeSpecName: "kube-api-access-xg57x") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "kube-api-access-xg57x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:44:24.106263 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.106241    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d764af95-ecf6-47b7-ba85-86896e32c61f" (UID: "d764af95-ecf6-47b7-ba85-86896e32c61f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:44:24.205271 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205209    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-cacerts\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.205271 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205234    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg57x\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-kube-api-access-xg57x\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.205271 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205249    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-csr-dns-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.205271 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205261    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d764af95-ecf6-47b7-ba85-86896e32c61f-local-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.205453 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205274    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.205453 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.205285    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d764af95-ecf6-47b7-ba85-86896e32c61f-istio-kubeconfig\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:44:24.630411 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.630374    2558 generic.go:356] "Generic (PLEG): container finished" podID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerID="f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858" exitCode=0
Jul 13 23:44:24.630817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.630431    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"
Jul 13 23:44:24.630817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.630454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" event={"ID":"d764af95-ecf6-47b7-ba85-86896e32c61f","Type":"ContainerDied","Data":"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"}
Jul 13 23:44:24.630817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.630505    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-szgzn" event={"ID":"d764af95-ecf6-47b7-ba85-86896e32c61f","Type":"ContainerDied","Data":"9110eabfa1d27bad15c6fd948cd1a16a91f35d8c19f746bb2d965fc6c8566a18"}
Jul 13 23:44:24.630817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.630525    2558 scope.go:117] "RemoveContainer" containerID="f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"
Jul 13 23:44:24.639047 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.639031    2558 scope.go:117] "RemoveContainer" containerID="f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"
Jul 13 23:44:24.639297 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:44:24.639280    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858\": container with ID starting with f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858 not found: ID does not exist" containerID="f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"
Jul 13 23:44:24.639343 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.639308    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858"} err="failed to get container status \"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858\": rpc error: code = NotFound desc = could not find container \"f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858\": container with ID starting with f18eac6032014da9aad0256c57659abcb71aa6f2bbb62f6303ce3bfe8bed2858 not found: ID does not exist"
Jul 13 23:44:24.659533 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.659511    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:24.672581 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:24.672560    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-szgzn"]
Jul 13 23:44:25.431147 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:25.431113    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" path="/var/lib/kubelet/pods/d764af95-ecf6-47b7-ba85-86896e32c61f/volumes"
Jul 13 23:44:28.606681 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:44:28.606649    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lq7qn"
Jul 13 23:45:13.686540 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.686507    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:13.686932 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.686781    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerName="discovery"
Jul 13 23:45:13.686932 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.686793    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerName="discovery"
Jul 13 23:45:13.686932 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.686838    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d764af95-ecf6-47b7-ba85-86896e32c61f" containerName="discovery"
Jul 13 23:45:13.689154 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.689137    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:13.692129 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.692091    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 23:45:13.692129 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.692107    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 23:45:13.692337 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.692167    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-brk2l\""
Jul 13 23:45:13.702743 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.702714    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:13.751328 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.751300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8df\" (UniqueName: \"kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df\") pod \"kuadrant-operator-catalog-mdvnt\" (UID: \"34e8afe8-e469-4160-a86d-9b267f6a3a99\") " pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:13.851832 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.851780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8df\" (UniqueName: \"kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df\") pod \"kuadrant-operator-catalog-mdvnt\" (UID: \"34e8afe8-e469-4160-a86d-9b267f6a3a99\") " pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:13.859560 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.859535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8df\" (UniqueName: \"kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df\") pod \"kuadrant-operator-catalog-mdvnt\" (UID: \"34e8afe8-e469-4160-a86d-9b267f6a3a99\") " pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:13.999105 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:13.999015    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:14.058766 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.058728    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:14.128472 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.128444    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:14.131443 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:14.131399    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e8afe8_e469_4160_a86d_9b267f6a3a99.slice/crio-fe4410e6f769fe27b48bee075edec826999f58a9d2560d41b10b04aa7eafa3aa WatchSource:0}: Error finding container fe4410e6f769fe27b48bee075edec826999f58a9d2560d41b10b04aa7eafa3aa: Status 404 returned error can't find the container with id fe4410e6f769fe27b48bee075edec826999f58a9d2560d41b10b04aa7eafa3aa
Jul 13 23:45:14.267586 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.267546    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hbhwf"]
Jul 13 23:45:14.270151 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.270135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:14.278011 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.277985    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hbhwf"]
Jul 13 23:45:14.355069 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.355032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfh8z\" (UniqueName: \"kubernetes.io/projected/76e3bf6e-608a-48f6-8768-e1f8d073e24d-kube-api-access-lfh8z\") pod \"kuadrant-operator-catalog-hbhwf\" (UID: \"76e3bf6e-608a-48f6-8768-e1f8d073e24d\") " pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:14.455803 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.455773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfh8z\" (UniqueName: \"kubernetes.io/projected/76e3bf6e-608a-48f6-8768-e1f8d073e24d-kube-api-access-lfh8z\") pod \"kuadrant-operator-catalog-hbhwf\" (UID: \"76e3bf6e-608a-48f6-8768-e1f8d073e24d\") " pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:14.468441 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.468410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfh8z\" (UniqueName: \"kubernetes.io/projected/76e3bf6e-608a-48f6-8768-e1f8d073e24d-kube-api-access-lfh8z\") pod \"kuadrant-operator-catalog-hbhwf\" (UID: \"76e3bf6e-608a-48f6-8768-e1f8d073e24d\") " pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:14.579977 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.579879    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:14.704716 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.704690    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hbhwf"]
Jul 13 23:45:14.706985 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:14.706962    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e3bf6e_608a_48f6_8768_e1f8d073e24d.slice/crio-bfea22b446164addeb3534db82d9f156e8e59c450c59c9804479857d5d3e918f WatchSource:0}: Error finding container bfea22b446164addeb3534db82d9f156e8e59c450c59c9804479857d5d3e918f: Status 404 returned error can't find the container with id bfea22b446164addeb3534db82d9f156e8e59c450c59c9804479857d5d3e918f
Jul 13 23:45:14.807946 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.807916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf" event={"ID":"76e3bf6e-608a-48f6-8768-e1f8d073e24d","Type":"ContainerStarted","Data":"bfea22b446164addeb3534db82d9f156e8e59c450c59c9804479857d5d3e918f"}
Jul 13 23:45:14.808970 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:14.808947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" event={"ID":"34e8afe8-e469-4160-a86d-9b267f6a3a99","Type":"ContainerStarted","Data":"fe4410e6f769fe27b48bee075edec826999f58a9d2560d41b10b04aa7eafa3aa"}
Jul 13 23:45:16.819753 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:16.819717    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf" event={"ID":"76e3bf6e-608a-48f6-8768-e1f8d073e24d","Type":"ContainerStarted","Data":"e48abb63fe1fc48698a92d78afd016eba330972a2427d21ecc19adc567f3d8e9"}
Jul 13 23:45:16.821302 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:16.821272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" event={"ID":"34e8afe8-e469-4160-a86d-9b267f6a3a99","Type":"ContainerStarted","Data":"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"}
Jul 13 23:45:16.821302 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:16.821289    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" podUID="34e8afe8-e469-4160-a86d-9b267f6a3a99" containerName="registry-server" containerID="cri-o://50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee" gracePeriod=2
Jul 13 23:45:16.836147 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:16.836103    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf" podStartSLOduration=1.05722844 podStartE2EDuration="2.836080942s" totalImagesPullingTime="1.778852502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:45:16.835866554 +0000 UTC m=+365.123084901" watchObservedRunningTime="2026-07-13 23:45:16.836080942 +0000 UTC m=+365.123299289"
Jul 13 23:45:16.850201 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:16.850161    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" podStartSLOduration=1.492140607 podStartE2EDuration="3.850146942s" totalImagesPullingTime="2.358006335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:45:16.84998418 +0000 UTC m=+365.137202527" watchObservedRunningTime="2026-07-13 23:45:16.850146942 +0000 UTC m=+365.137365286"
Jul 13 23:45:17.058158 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.058132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:17.081166 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.081096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p8df\" (UniqueName: \"kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df\") pod \"34e8afe8-e469-4160-a86d-9b267f6a3a99\" (UID: \"34e8afe8-e469-4160-a86d-9b267f6a3a99\") "
Jul 13 23:45:17.083143 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.083118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df" (OuterVolumeSpecName: "kube-api-access-4p8df") pod "34e8afe8-e469-4160-a86d-9b267f6a3a99" (UID: "34e8afe8-e469-4160-a86d-9b267f6a3a99"). InnerVolumeSpecName "kube-api-access-4p8df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:45:17.182577 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.182545    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8df\" (UniqueName: \"kubernetes.io/projected/34e8afe8-e469-4160-a86d-9b267f6a3a99-kube-api-access-4p8df\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:45:17.826297 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.826263    2558 generic.go:356] "Generic (PLEG): container finished" podID="34e8afe8-e469-4160-a86d-9b267f6a3a99" containerID="50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee" exitCode=0
Jul 13 23:45:17.826692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.826305    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt"
Jul 13 23:45:17.826692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.826351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" event={"ID":"34e8afe8-e469-4160-a86d-9b267f6a3a99","Type":"ContainerDied","Data":"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"}
Jul 13 23:45:17.826692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.826397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-mdvnt" event={"ID":"34e8afe8-e469-4160-a86d-9b267f6a3a99","Type":"ContainerDied","Data":"fe4410e6f769fe27b48bee075edec826999f58a9d2560d41b10b04aa7eafa3aa"}
Jul 13 23:45:17.826692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.826416    2558 scope.go:117] "RemoveContainer" containerID="50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"
Jul 13 23:45:17.834860 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.834841    2558 scope.go:117] "RemoveContainer" containerID="50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"
Jul 13 23:45:17.835146 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:45:17.835129    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee\": container with ID starting with 50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee not found: ID does not exist" containerID="50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"
Jul 13 23:45:17.835200 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.835155    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee"} err="failed to get container status \"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee\": rpc error: code = NotFound desc = could not find container \"50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee\": container with ID starting with 50d2e3c1579774edf24b249ddaff42fe5b4d881d5efc11a655cd346bd4c57bee not found: ID does not exist"
Jul 13 23:45:17.842714 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.842691    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:17.844331 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:17.844312    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-mdvnt"]
Jul 13 23:45:19.430700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:19.430658    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e8afe8-e469-4160-a86d-9b267f6a3a99" path="/var/lib/kubelet/pods/34e8afe8-e469-4160-a86d-9b267f6a3a99/volumes"
Jul 13 23:45:24.580618 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:24.580581    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:24.581066 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:24.580653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:24.602761 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:24.602739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:24.876164 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:24.876092    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hbhwf"
Jul 13 23:45:45.655700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.655668    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"]
Jul 13 23:45:45.656190 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.655939    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e8afe8-e469-4160-a86d-9b267f6a3a99" containerName="registry-server"
Jul 13 23:45:45.656190 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.655952    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e8afe8-e469-4160-a86d-9b267f6a3a99" containerName="registry-server"
Jul 13 23:45:45.656190 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.655999    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e8afe8-e469-4160-a86d-9b267f6a3a99" containerName="registry-server"
Jul 13 23:45:45.658784 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.658766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:45.662320 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.662292    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lmb22\""
Jul 13 23:45:45.668706 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.668683    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"]
Jul 13 23:45:45.797946 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.797913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvmx\" (UniqueName: \"kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx\") pod \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" (UID: \"04f68524-93be-4ec6-9a2a-9202c362a4b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:45.898758 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.898712    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvmx\" (UniqueName: \"kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx\") pod \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" (UID: \"04f68524-93be-4ec6-9a2a-9202c362a4b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:45.913286 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.913215    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvmx\" (UniqueName: \"kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx\") pod \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" (UID: \"04f68524-93be-4ec6-9a2a-9202c362a4b4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:45.968812 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:45.968782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:46.101375 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:46.101353    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"]
Jul 13 23:45:46.103874 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:46.103849    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f68524_93be_4ec6_9a2a_9202c362a4b4.slice/crio-4a7390ab4c21552ddee5606d1d94827417343c56fd064e10bc4e8fc64fccd5fc WatchSource:0}: Error finding container 4a7390ab4c21552ddee5606d1d94827417343c56fd064e10bc4e8fc64fccd5fc: Status 404 returned error can't find the container with id 4a7390ab4c21552ddee5606d1d94827417343c56fd064e10bc4e8fc64fccd5fc
Jul 13 23:45:46.943824 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:46.943722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp" event={"ID":"04f68524-93be-4ec6-9a2a-9202c362a4b4","Type":"ContainerStarted","Data":"4a7390ab4c21552ddee5606d1d94827417343c56fd064e10bc4e8fc64fccd5fc"}
Jul 13 23:45:48.958222 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:48.958187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp" event={"ID":"04f68524-93be-4ec6-9a2a-9202c362a4b4","Type":"ContainerStarted","Data":"a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4"}
Jul 13 23:45:48.958597 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:48.958322    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:45:48.977169 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:48.977129    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp" podStartSLOduration=2.164484342 podStartE2EDuration="3.977115503s" totalImagesPullingTime="1.812631161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:45:48.976116059 +0000 UTC m=+397.263334418" watchObservedRunningTime="2026-07-13 23:45:48.977115503 +0000 UTC m=+397.264333848"
Jul 13 23:45:50.801403 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.801369    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"]
Jul 13 23:45:50.809160 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.809141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:50.812444 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.812426    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r24xl\""
Jul 13 23:45:50.822437 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.822415    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"]
Jul 13 23:45:50.939851 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.939821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:50.939986 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:50.939860    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mflx\" (UniqueName: \"kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.040474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.040439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.040569 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.040481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mflx\" (UniqueName: \"kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.040913 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.040873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.048674 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.048652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mflx\" (UniqueName: \"kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.119174 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.119123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:51.251910 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.251737    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"]
Jul 13 23:45:51.254649 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:51.254622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8dd017_8133_4a01_9391_d46d071d346b.slice/crio-ef3ed5d4a26e43f16cb67ebf66cfcfe3efd21792a6214c37b035c91a5b5e7888 WatchSource:0}: Error finding container ef3ed5d4a26e43f16cb67ebf66cfcfe3efd21792a6214c37b035c91a5b5e7888: Status 404 returned error can't find the container with id ef3ed5d4a26e43f16cb67ebf66cfcfe3efd21792a6214c37b035c91a5b5e7888
Jul 13 23:45:51.971460 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:51.971426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66" event={"ID":"2b8dd017-8133-4a01-9391-d46d071d346b","Type":"ContainerStarted","Data":"ef3ed5d4a26e43f16cb67ebf66cfcfe3efd21792a6214c37b035c91a5b5e7888"}
Jul 13 23:45:53.483197 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.483169    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"]
Jul 13 23:45:53.486446 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.486431    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:53.488898 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.488861    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x5stz\""
Jul 13 23:45:53.494428 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.494406    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"]
Jul 13 23:45:53.558524 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.558501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rk2\" (UniqueName: \"kubernetes.io/projected/9c0db5e9-d628-4ebb-a184-f9083b4fce4b-kube-api-access-68rk2\") pod \"authorino-operator-75cc8d5cb7-pgkl9\" (UID: \"9c0db5e9-d628-4ebb-a184-f9083b4fce4b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:53.659226 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.659201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rk2\" (UniqueName: \"kubernetes.io/projected/9c0db5e9-d628-4ebb-a184-f9083b4fce4b-kube-api-access-68rk2\") pod \"authorino-operator-75cc8d5cb7-pgkl9\" (UID: \"9c0db5e9-d628-4ebb-a184-f9083b4fce4b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:53.673366 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.673343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rk2\" (UniqueName: \"kubernetes.io/projected/9c0db5e9-d628-4ebb-a184-f9083b4fce4b-kube-api-access-68rk2\") pod \"authorino-operator-75cc8d5cb7-pgkl9\" (UID: \"9c0db5e9-d628-4ebb-a184-f9083b4fce4b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:53.796985 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.796956    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:53.936901 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.936812    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"]
Jul 13 23:45:53.939686 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:53.939658    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0db5e9_d628_4ebb_a184_f9083b4fce4b.slice/crio-5a3a414d4a50b6880647e08a5fa838025bb53d6e68e64943849d5e91f096d8ad WatchSource:0}: Error finding container 5a3a414d4a50b6880647e08a5fa838025bb53d6e68e64943849d5e91f096d8ad: Status 404 returned error can't find the container with id 5a3a414d4a50b6880647e08a5fa838025bb53d6e68e64943849d5e91f096d8ad
Jul 13 23:45:53.980248 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:53.980212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9" event={"ID":"9c0db5e9-d628-4ebb-a184-f9083b4fce4b","Type":"ContainerStarted","Data":"5a3a414d4a50b6880647e08a5fa838025bb53d6e68e64943849d5e91f096d8ad"}
Jul 13 23:45:56.830935 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.830881    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"]
Jul 13 23:45:56.834307 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.834287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:45:56.838220 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.838197    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 13 23:45:56.838349 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.838304    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-d2v2k\""
Jul 13 23:45:56.848386 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.848360    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"]
Jul 13 23:45:56.985527 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:56.985494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw52w\" (UniqueName: \"kubernetes.io/projected/fd6f8026-bd7c-426b-b21a-3cfc5bf1f289-kube-api-access-qw52w\") pod \"dns-operator-controller-manager-864dd78c4c-fmc6g\" (UID: \"fd6f8026-bd7c-426b-b21a-3cfc5bf1f289\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:45:57.086350 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.086260    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw52w\" (UniqueName: \"kubernetes.io/projected/fd6f8026-bd7c-426b-b21a-3cfc5bf1f289-kube-api-access-qw52w\") pod \"dns-operator-controller-manager-864dd78c4c-fmc6g\" (UID: \"fd6f8026-bd7c-426b-b21a-3cfc5bf1f289\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:45:57.102279 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.102249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw52w\" (UniqueName: \"kubernetes.io/projected/fd6f8026-bd7c-426b-b21a-3cfc5bf1f289-kube-api-access-qw52w\") pod \"dns-operator-controller-manager-864dd78c4c-fmc6g\" (UID: \"fd6f8026-bd7c-426b-b21a-3cfc5bf1f289\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:45:57.148032 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.147987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:45:57.343765 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.343741    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"]
Jul 13 23:45:57.345534 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:45:57.345506    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6f8026_bd7c_426b_b21a_3cfc5bf1f289.slice/crio-039511078c23f1d31828eadea2a8de8102d3aa428026f20fe04dfae4a7db7bac WatchSource:0}: Error finding container 039511078c23f1d31828eadea2a8de8102d3aa428026f20fe04dfae4a7db7bac: Status 404 returned error can't find the container with id 039511078c23f1d31828eadea2a8de8102d3aa428026f20fe04dfae4a7db7bac
Jul 13 23:45:57.998650 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.998614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9" event={"ID":"9c0db5e9-d628-4ebb-a184-f9083b4fce4b","Type":"ContainerStarted","Data":"17f18e5397ec5862f67b83331d58a60d618e553764e7191ac3c4a5e448cfbe2d"}
Jul 13 23:45:57.999067 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.998737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:45:57.999912 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:57.999866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g" event={"ID":"fd6f8026-bd7c-426b-b21a-3cfc5bf1f289","Type":"ContainerStarted","Data":"039511078c23f1d31828eadea2a8de8102d3aa428026f20fe04dfae4a7db7bac"}
Jul 13 23:45:58.001402 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:58.001380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66" event={"ID":"2b8dd017-8133-4a01-9391-d46d071d346b","Type":"ContainerStarted","Data":"40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0"}
Jul 13 23:45:58.001573 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:58.001520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:45:58.015080 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:58.015039    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9" podStartSLOduration=1.737992548 podStartE2EDuration="5.015021063s" totalImagesPullingTime="3.277028515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:45:58.014574897 +0000 UTC m=+406.301793245" watchObservedRunningTime="2026-07-13 23:45:58.015021063 +0000 UTC m=+406.302239412"
Jul 13 23:45:58.035830 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:58.035783    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66" podStartSLOduration=2.072123775 podStartE2EDuration="8.035768652s" totalImagesPullingTime="5.963644877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:45:58.0336547 +0000 UTC m=+406.320873047" watchObservedRunningTime="2026-07-13 23:45:58.035768652 +0000 UTC m=+406.322986997"
Jul 13 23:45:59.964815 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:45:59.964734    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:00.013503 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.013467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g" event={"ID":"fd6f8026-bd7c-426b-b21a-3cfc5bf1f289","Type":"ContainerStarted","Data":"386a298de5d829af74165b21d6636c377af942901e70a3323c55d86d617c59b8"}
Jul 13 23:46:00.013680 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.013602    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:46:00.037006 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.036953    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g" podStartSLOduration=1.849804389 podStartE2EDuration="4.036935545s" totalImagesPullingTime="2.187131156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:00.034868412 +0000 UTC m=+408.322086760" watchObservedRunningTime="2026-07-13 23:46:00.036935545 +0000 UTC m=+408.324153891"
Jul 13 23:46:00.080237 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.080205    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"]
Jul 13 23:46:00.083550 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.083530    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.087132 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.087110    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gbt4b\""
Jul 13 23:46:00.087234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.087109    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 13 23:46:00.087234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.087134    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 13 23:46:00.094302 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.094282    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"]
Jul 13 23:46:00.215314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.215238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.215314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.215277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/24a72699-6483-49f1-a93f-1ed954e2d3a0-kube-api-access-n8zvq\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.215314 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.215298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24a72699-6483-49f1-a93f-1ed954e2d3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.316544 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.316513    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.316677 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.316556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/24a72699-6483-49f1-a93f-1ed954e2d3a0-kube-api-access-n8zvq\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.316677 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.316584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24a72699-6483-49f1-a93f-1ed954e2d3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.316677 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:00.316648    2558 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 13 23:46:00.316823 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:00.316708    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert podName:24a72699-6483-49f1-a93f-1ed954e2d3a0 nodeName:}" failed. No retries permitted until 2026-07-13 23:46:00.816692376 +0000 UTC m=+409.103910700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-qlmps" (UID: "24a72699-6483-49f1-a93f-1ed954e2d3a0") : secret "plugin-serving-cert" not found
Jul 13 23:46:00.317242 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.317220    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/24a72699-6483-49f1-a93f-1ed954e2d3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.329276 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.329257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zvq\" (UniqueName: \"kubernetes.io/projected/24a72699-6483-49f1-a93f-1ed954e2d3a0-kube-api-access-n8zvq\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.820173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.820127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.822493 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.822461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/24a72699-6483-49f1-a93f-1ed954e2d3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qlmps\" (UID: \"24a72699-6483-49f1-a93f-1ed954e2d3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:00.993301 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:00.993270    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"
Jul 13 23:46:01.145846 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:01.145820    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps"]
Jul 13 23:46:01.147694 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:01.147667    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24a72699_6483_49f1_a93f_1ed954e2d3a0.slice/crio-114786ffb1103e9c609d5c0ccc3237bad899adbac37aa152a1c1db9ba71adf87 WatchSource:0}: Error finding container 114786ffb1103e9c609d5c0ccc3237bad899adbac37aa152a1c1db9ba71adf87: Status 404 returned error can't find the container with id 114786ffb1103e9c609d5c0ccc3237bad899adbac37aa152a1c1db9ba71adf87
Jul 13 23:46:02.024648 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:02.024610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps" event={"ID":"24a72699-6483-49f1-a93f-1ed954e2d3a0","Type":"ContainerStarted","Data":"114786ffb1103e9c609d5c0ccc3237bad899adbac37aa152a1c1db9ba71adf87"}
Jul 13 23:46:09.008469 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:09.008434    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pgkl9"
Jul 13 23:46:09.008948 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:09.008509    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:10.592192 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.592153    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"]
Jul 13 23:46:10.613294 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.613258    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.615024 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.614972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"]
Jul 13 23:46:10.697741 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.697697    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"]
Jul 13 23:46:10.698025 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.697992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.698172 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.698056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bg8\" (UniqueName: \"kubernetes.io/projected/ee719676-645d-450b-9219-0872b023cb8a-kube-api-access-r2bg8\") pod \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.698221 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.698195    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-r2bg8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j" podUID="ee719676-645d-450b-9219-0872b023cb8a"
Jul 13 23:46:10.729436 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.729406    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"]
Jul 13 23:46:10.736702 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.736679    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.749239 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.749216    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"]
Jul 13 23:46:10.784001 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.783964    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"]
Jul 13 23:46:10.795175 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.795142    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"]
Jul 13 23:46:10.798617 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.798584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjnj\" (UniqueName: \"kubernetes.io/projected/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-kube-api-access-zkjnj\") pod \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.798745 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.798664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bg8\" (UniqueName: \"kubernetes.io/projected/ee719676-645d-450b-9219-0872b023cb8a-kube-api-access-r2bg8\") pod \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.798745 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.798702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.798865 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.798820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.799237 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.799216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:10.804878 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.804853    2558 projected.go:196] Error preparing data for projected volume kube-api-access-r2bg8 for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-2mr6j" not found
Jul 13 23:46:10.805008 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.804951    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee719676-645d-450b-9219-0872b023cb8a-kube-api-access-r2bg8 podName:ee719676-645d-450b-9219-0872b023cb8a nodeName:}" failed. No retries permitted until 2026-07-13 23:46:11.304930435 +0000 UTC m=+419.592148759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r2bg8" (UniqueName: "kubernetes.io/projected/ee719676-645d-450b-9219-0872b023cb8a-kube-api-access-r2bg8") pod "kuadrant-operator-controller-manager-55467d9f7-2mr6j" (UID: "ee719676-645d-450b-9219-0872b023cb8a") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-2mr6j" not found
Jul 13 23:46:10.806397 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.806373    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"]
Jul 13 23:46:10.806599 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.806573    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" containerName="manager" containerID="cri-o://40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0" gracePeriod=2
Jul 13 23:46:10.819149 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.819116    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"]
Jul 13 23:46:10.828150 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.828114    2558 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2b8dd017-8133-4a01-9391-d46d071d346b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-c4t66\": pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:10.836681 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.836650    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"]
Jul 13 23:46:10.836981 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.836950    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-zkjnj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e"
Jul 13 23:46:10.846353 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.846262    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:10.847067 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.847041    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" containerName="manager"
Jul 13 23:46:10.847067 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.847067    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" containerName="manager"
Jul 13 23:46:10.847213 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.847159    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" containerName="manager"
Jul 13 23:46:10.850536 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.850514    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"]
Jul 13 23:46:10.850973 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.850739    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:10.871682 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.871652    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:10.873879 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.873491    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:46:10.877174 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.877151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"]
Jul 13 23:46:10.877330 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.877276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:10.877396 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.877343    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" containerName="manager" containerID="cri-o://a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4" gracePeriod=2
Jul 13 23:46:10.882165 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.882113    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"]
Jul 13 23:46:10.895752 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.895575    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:10.896810 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.896790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:46:10.898093 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.898068    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:10.899578 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899552    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjnj\" (UniqueName: \"kubernetes.io/projected/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-kube-api-access-zkjnj\") pod \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.899663 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrvg\" (UniqueName: \"kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:10.899731 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.899731 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:10.900475 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899746    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7p8c\" (UniqueName: \"kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:10.900475 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.899804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:10.900475 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.900220    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:10.902286 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.902266    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"]
Jul 13 23:46:10.902698 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.902677    2558 projected.go:196] Error preparing data for projected volume kube-api-access-zkjnj for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-b4c4w" not found
Jul 13 23:46:10.902777 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.902755    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-kube-api-access-zkjnj podName:34d2b8e5-6fd0-44a3-9908-882f6867dd1e nodeName:}" failed. No retries permitted until 2026-07-13 23:46:11.402734708 +0000 UTC m=+419.689953046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zkjnj" (UniqueName: "kubernetes.io/projected/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-kube-api-access-zkjnj") pod "kuadrant-operator-controller-manager-86878c769f-b4c4w" (UID: "34d2b8e5-6fd0-44a3-9908-882f6867dd1e") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-b4c4w" not found
Jul 13 23:46:10.902978 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.902957    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" containerName="manager"
Jul 13 23:46:10.902978 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.902980    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" containerName="manager"
Jul 13 23:46:10.903133 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.903071    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" containerName="manager"
Jul 13 23:46:10.906467 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.906439    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:10.921284 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:10.921262    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"]
Jul 13 23:46:10.930482 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.930454    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:10.932951 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.932802    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:10.935207 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:10.935181    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:11.000626 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.000592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpgn\" (UniqueName: \"kubernetes.io/projected/d213ae4e-650d-4ad4-9a92-5822ddabdf84-kube-api-access-9zpgn\") pod \"limitador-operator-controller-manager-859d7b78bd-8b84l\" (UID: \"d213ae4e-650d-4ad4-9a92-5822ddabdf84\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:11.000828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.000807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrvg\" (UniqueName: \"kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:11.000949 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.000930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:11.001009 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.000988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7p8c\" (UniqueName: \"kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:11.001081 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.001048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:11.001320 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.001291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:11.001427 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.001407    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:11.009959 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.009938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7p8c\" (UniqueName: \"kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-csmq2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:11.010113 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.010094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrvg\" (UniqueName: \"kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg\") pod \"kuadrant-operator-controller-manager-86878c769f-78blz\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:11.021360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.021312    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fmc6g"
Jul 13 23:46:11.023743 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.023710    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:11.044757 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.044726    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:11.071562 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.071539    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:11.071562 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.071553    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:11.074035 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.074004    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:11.076043 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.076008    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:11.077920 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.077873    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:11.079835 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.079797    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:11.081782 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.081744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:11.082220 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.082194    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:11.084165 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.084128    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:11.084260 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.084204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:11.086090 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.086060    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="ee719676-645d-450b-9219-0872b023cb8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:11.088145 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.088117    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:11.090177 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.090138    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:11.092472 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.092349    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:11.094280 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:11.094251    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="ee719676-645d-450b-9219-0872b023cb8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:11.102057 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.101807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpgn\" (UniqueName: \"kubernetes.io/projected/d213ae4e-650d-4ad4-9a92-5822ddabdf84-kube-api-access-9zpgn\") pod \"limitador-operator-controller-manager-859d7b78bd-8b84l\" (UID: \"d213ae4e-650d-4ad4-9a92-5822ddabdf84\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:11.102057 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.101989    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bg8\" (UniqueName: \"kubernetes.io/projected/ee719676-645d-450b-9219-0872b023cb8a-kube-api-access-r2bg8\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:11.102057 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.102007    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjnj\" (UniqueName: \"kubernetes.io/projected/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-kube-api-access-zkjnj\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:11.114686 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.114649    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpgn\" (UniqueName: \"kubernetes.io/projected/d213ae4e-650d-4ad4-9a92-5822ddabdf84-kube-api-access-9zpgn\") pod \"limitador-operator-controller-manager-859d7b78bd-8b84l\" (UID: \"d213ae4e-650d-4ad4-9a92-5822ddabdf84\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:11.181441 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.181409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:11.203331 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.203306    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume\") pod \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\" (UID: \"34d2b8e5-6fd0-44a3-9908-882f6867dd1e\") "
Jul 13 23:46:11.203458 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.203372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume\") pod \"ee719676-645d-450b-9219-0872b023cb8a\" (UID: \"ee719676-645d-450b-9219-0872b023cb8a\") "
Jul 13 23:46:11.203649 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.203598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "34d2b8e5-6fd0-44a3-9908-882f6867dd1e" (UID: "34d2b8e5-6fd0-44a3-9908-882f6867dd1e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:46:11.203764 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.203654    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee719676-645d-450b-9219-0872b023cb8a" (UID: "ee719676-645d-450b-9219-0872b023cb8a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:46:11.234999 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.234970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:11.243651 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.243626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:11.304443 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.304417    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34d2b8e5-6fd0-44a3-9908-882f6867dd1e-extensions-socket-volume\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:11.304443 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.304442    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee719676-645d-450b-9219-0872b023cb8a-extensions-socket-volume\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:11.432986 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.432909    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" path="/var/lib/kubelet/pods/34d2b8e5-6fd0-44a3-9908-882f6867dd1e/volumes"
Jul 13 23:46:11.433208 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:11.433194    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee719676-645d-450b-9219-0872b023cb8a" path="/var/lib/kubelet/pods/ee719676-645d-450b-9219-0872b023cb8a/volumes"
Jul 13 23:46:12.075678 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:12.075646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:12.076138 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:12.075646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:12.082581 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:12.082542    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="ee719676-645d-450b-9219-0872b023cb8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:12.084510 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:12.084486    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:12.086383 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:12.086356    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="ee719676-645d-450b-9219-0872b023cb8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:13.430401 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:13.430348    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-b4c4w\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="34d2b8e5-6fd0-44a3-9908-882f6867dd1e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b4c4w"
Jul 13 23:46:13.432507 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:13.432456    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-2mr6j\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="ee719676-645d-450b-9219-0872b023cb8a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-2mr6j"
Jul 13 23:46:22.286665 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.286636    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:22.288926 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:22.288877    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:22.302193 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.302170    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:22.304496 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:22.304404    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:22.306172 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:22.306143    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:22.339977 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.339950    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:46:22.342261 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:22.342226    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc6c7f_0e02_407e_a28c_64087b0ed06e.slice/crio-93268a21a0a105dc910db686be47397187d212fae8767128cc1878ceec9dce2a WatchSource:0}: Error finding container 93268a21a0a105dc910db686be47397187d212fae8767128cc1878ceec9dce2a: Status 404 returned error can't find the container with id 93268a21a0a105dc910db686be47397187d212fae8767128cc1878ceec9dce2a
Jul 13 23:46:22.359539 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.359519    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"]
Jul 13 23:46:22.362586 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:22.362559    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd213ae4e_650d_4ad4_9a92_5822ddabdf84.slice/crio-45233e459e9b467ce9d1f27c4a690072985b47c4a09b7f4ff2f46ad70dfde974 WatchSource:0}: Error finding container 45233e459e9b467ce9d1f27c4a690072985b47c4a09b7f4ff2f46ad70dfde974: Status 404 returned error can't find the container with id 45233e459e9b467ce9d1f27c4a690072985b47c4a09b7f4ff2f46ad70dfde974
Jul 13 23:46:22.377395 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.377369    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:22.380250 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:22.380224    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cf04cb_d20c_46b8_9a45_52486c4087a2.slice/crio-8508045d5a17eee6b5a7808d99aa245a3c8430fb048e49898e9c835a0de9cf7a WatchSource:0}: Error finding container 8508045d5a17eee6b5a7808d99aa245a3c8430fb048e49898e9c835a0de9cf7a: Status 404 returned error can't find the container with id 8508045d5a17eee6b5a7808d99aa245a3c8430fb048e49898e9c835a0de9cf7a
Jul 13 23:46:22.401772 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.401750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvvmx\" (UniqueName: \"kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx\") pod \"04f68524-93be-4ec6-9a2a-9202c362a4b4\" (UID: \"04f68524-93be-4ec6-9a2a-9202c362a4b4\") "
Jul 13 23:46:22.401873 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.401858    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mflx\" (UniqueName: \"kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx\") pod \"2b8dd017-8133-4a01-9391-d46d071d346b\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") "
Jul 13 23:46:22.401952 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.401922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume\") pod \"2b8dd017-8133-4a01-9391-d46d071d346b\" (UID: \"2b8dd017-8133-4a01-9391-d46d071d346b\") "
Jul 13 23:46:22.402435 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.402406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2b8dd017-8133-4a01-9391-d46d071d346b" (UID: "2b8dd017-8133-4a01-9391-d46d071d346b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:46:22.403996 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.403953    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx" (OuterVolumeSpecName: "kube-api-access-rvvmx") pod "04f68524-93be-4ec6-9a2a-9202c362a4b4" (UID: "04f68524-93be-4ec6-9a2a-9202c362a4b4"). InnerVolumeSpecName "kube-api-access-rvvmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:46:22.404075 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.403970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx" (OuterVolumeSpecName: "kube-api-access-6mflx") pod "2b8dd017-8133-4a01-9391-d46d071d346b" (UID: "2b8dd017-8133-4a01-9391-d46d071d346b"). InnerVolumeSpecName "kube-api-access-6mflx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:46:22.502817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.502787    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mflx\" (UniqueName: \"kubernetes.io/projected/2b8dd017-8133-4a01-9391-d46d071d346b-kube-api-access-6mflx\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:22.502817 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.502815    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b8dd017-8133-4a01-9391-d46d071d346b-extensions-socket-volume\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:22.503056 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:22.502829    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvmx\" (UniqueName: \"kubernetes.io/projected/04f68524-93be-4ec6-9a2a-9202c362a4b4-kube-api-access-rvvmx\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:23.132568 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.132468    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l" event={"ID":"d213ae4e-650d-4ad4-9a92-5822ddabdf84","Type":"ContainerStarted","Data":"3dcde9ee0d7de96452e5e94ae32d4598881d3d22d5ee59101d1b8be4cc350ca5"}
Jul 13 23:46:23.132568 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.132522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l" event={"ID":"d213ae4e-650d-4ad4-9a92-5822ddabdf84","Type":"ContainerStarted","Data":"45233e459e9b467ce9d1f27c4a690072985b47c4a09b7f4ff2f46ad70dfde974"}
Jul 13 23:46:23.132757 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.132571    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:23.134698 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.134589    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" event={"ID":"15cf04cb-d20c-46b8-9a45-52486c4087a2","Type":"ContainerStarted","Data":"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"}
Jul 13 23:46:23.134698 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.134663    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:23.135110 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.134708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" event={"ID":"15cf04cb-d20c-46b8-9a45-52486c4087a2","Type":"ContainerStarted","Data":"8508045d5a17eee6b5a7808d99aa245a3c8430fb048e49898e9c835a0de9cf7a"}
Jul 13 23:46:23.135110 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.134928    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:23.136474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.136448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" event={"ID":"9ffc6c7f-0e02-407e-a28c-64087b0ed06e","Type":"ContainerStarted","Data":"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"}
Jul 13 23:46:23.136613 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.136496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" event={"ID":"9ffc6c7f-0e02-407e-a28c-64087b0ed06e","Type":"ContainerStarted","Data":"93268a21a0a105dc910db686be47397187d212fae8767128cc1878ceec9dce2a"}
Jul 13 23:46:23.136613 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.136549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:23.136726 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.136669    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:23.137828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.137806    2558 generic.go:356] "Generic (PLEG): container finished" podID="04f68524-93be-4ec6-9a2a-9202c362a4b4" containerID="a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4" exitCode=0
Jul 13 23:46:23.137940 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.137840    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:23.137940 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.137928    2558 scope.go:117] "RemoveContainer" containerID="a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4"
Jul 13 23:46:23.139684 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.139664    2558 generic.go:356] "Generic (PLEG): container finished" podID="2b8dd017-8133-4a01-9391-d46d071d346b" containerID="40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0" exitCode=0
Jul 13 23:46:23.139796 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.139694    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:23.141818 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.141794    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps" event={"ID":"24a72699-6483-49f1-a93f-1ed954e2d3a0","Type":"ContainerStarted","Data":"b80181058eb20bcbc3d4122119891cec4c2f537d88b3d0ebdc7ced18a131882a"}
Jul 13 23:46:23.147608 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.147587    2558 scope.go:117] "RemoveContainer" containerID="a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4"
Jul 13 23:46:23.147879 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.147855    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4\": container with ID starting with a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4 not found: ID does not exist" containerID="a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4"
Jul 13 23:46:23.147987 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.147964    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4"} err="failed to get container status \"a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4\": rpc error: code = NotFound desc = could not find container \"a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4\": container with ID starting with a0a1acf4e1c6b0426718e9cde939eea912aa235cbdb2e9a07277893b12ec2fd4 not found: ID does not exist"
Jul 13 23:46:23.148049 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.147990    2558 scope.go:117] "RemoveContainer" containerID="40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0"
Jul 13 23:46:23.155291 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.155261    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:23.155457 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.155415    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l" podStartSLOduration=13.155399104 podStartE2EDuration="13.155399104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:23.153260826 +0000 UTC m=+431.440479167" watchObservedRunningTime="2026-07-13 23:46:23.155399104 +0000 UTC m=+431.442617450"
Jul 13 23:46:23.156385 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.156369    2558 scope.go:117] "RemoveContainer" containerID="40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0"
Jul 13 23:46:23.156605 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.156585    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0\": container with ID starting with 40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0 not found: ID does not exist" containerID="40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0"
Jul 13 23:46:23.156659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.156613    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0"} err="failed to get container status \"40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0\": rpc error: code = NotFound desc = could not find container \"40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0\": container with ID starting with 40fd54d0d5d782779952bbe9168f56f60e3726c9e51b677eaa97f3c490447dd0 not found: ID does not exist"
Jul 13 23:46:23.177389 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.177350    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" podStartSLOduration=13.1773354 podStartE2EDuration="13.1773354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:23.175486176 +0000 UTC m=+431.462704521" watchObservedRunningTime="2026-07-13 23:46:23.1773354 +0000 UTC m=+431.464553747"
Jul 13 23:46:23.190143 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.190109    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qlmps" podStartSLOduration=2.12192609 podStartE2EDuration="23.190096245s" totalImagesPullingTime="21.068170155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:23.189233966 +0000 UTC m=+431.476452339" watchObservedRunningTime="2026-07-13 23:46:23.190096245 +0000 UTC m=+431.477314591"
Jul 13 23:46:23.207466 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.207429    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" podStartSLOduration=13.20741802 podStartE2EDuration="13.20741802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:23.205988255 +0000 UTC m=+431.493206614" watchObservedRunningTime="2026-07-13 23:46:23.20741802 +0000 UTC m=+431.494636428"
Jul 13 23:46:23.207916 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.207867    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:23.429823 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.429727    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gvdsp\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gvdsp"
Jul 13 23:46:23.431583 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:23.431544    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4t66\" is forbidden: User \"system:node:ip-10-0-128-174.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-174.ec2.internal' and this object" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4t66"
Jul 13 23:46:23.436079 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.436052    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04f68524-93be-4ec6-9a2a-9202c362a4b4" path="/var/lib/kubelet/pods/04f68524-93be-4ec6-9a2a-9202c362a4b4/volumes"
Jul 13 23:46:23.436471 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:23.436454    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b8dd017-8133-4a01-9391-d46d071d346b" path="/var/lib/kubelet/pods/2b8dd017-8133-4a01-9391-d46d071d346b/volumes"
Jul 13 23:46:34.150008 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.149931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:34.150008 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.150002    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8b84l"
Jul 13 23:46:34.150449 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.150034    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:46:34.247242 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.247209    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:34.247451 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.247429    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" podUID="15cf04cb-d20c-46b8-9a45-52486c4087a2" containerName="manager" containerID="cri-o://bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb" gracePeriod=10
Jul 13 23:46:34.490660 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.490636    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:34.594544 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.594518    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7p8c\" (UniqueName: \"kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c\") pod \"15cf04cb-d20c-46b8-9a45-52486c4087a2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") "
Jul 13 23:46:34.594683 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.594562    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume\") pod \"15cf04cb-d20c-46b8-9a45-52486c4087a2\" (UID: \"15cf04cb-d20c-46b8-9a45-52486c4087a2\") "
Jul 13 23:46:34.595052 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.595024    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "15cf04cb-d20c-46b8-9a45-52486c4087a2" (UID: "15cf04cb-d20c-46b8-9a45-52486c4087a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:46:34.597041 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.597015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c" (OuterVolumeSpecName: "kube-api-access-f7p8c") pod "15cf04cb-d20c-46b8-9a45-52486c4087a2" (UID: "15cf04cb-d20c-46b8-9a45-52486c4087a2"). InnerVolumeSpecName "kube-api-access-f7p8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:46:34.695743 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.695688    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7p8c\" (UniqueName: \"kubernetes.io/projected/15cf04cb-d20c-46b8-9a45-52486c4087a2-kube-api-access-f7p8c\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:34.695743 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:34.695707    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15cf04cb-d20c-46b8-9a45-52486c4087a2-extensions-socket-volume\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:46:35.193899 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.193866    2558 generic.go:356] "Generic (PLEG): container finished" podID="15cf04cb-d20c-46b8-9a45-52486c4087a2" containerID="bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb" exitCode=0
Jul 13 23:46:35.194304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.193923    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"
Jul 13 23:46:35.194304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.193926    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" event={"ID":"15cf04cb-d20c-46b8-9a45-52486c4087a2","Type":"ContainerDied","Data":"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"}
Jul 13 23:46:35.194304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.194083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2" event={"ID":"15cf04cb-d20c-46b8-9a45-52486c4087a2","Type":"ContainerDied","Data":"8508045d5a17eee6b5a7808d99aa245a3c8430fb048e49898e9c835a0de9cf7a"}
Jul 13 23:46:35.194304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.194115    2558 scope.go:117] "RemoveContainer" containerID="bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"
Jul 13 23:46:35.202218 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.202199    2558 scope.go:117] "RemoveContainer" containerID="bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"
Jul 13 23:46:35.202446 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:46:35.202429    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb\": container with ID starting with bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb not found: ID does not exist" containerID="bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"
Jul 13 23:46:35.202489 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.202454    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb"} err="failed to get container status \"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb\": rpc error: code = NotFound desc = could not find container \"bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb\": container with ID starting with bbcdb39f98a4a4dedeb5054d250c2f2123da5facef84e0b8e5459387eac9afcb not found: ID does not exist"
Jul 13 23:46:35.216980 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.216957    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:35.222592 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.222574    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-csmq2"]
Jul 13 23:46:35.431547 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:35.431521    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15cf04cb-d20c-46b8-9a45-52486c4087a2" path="/var/lib/kubelet/pods/15cf04cb-d20c-46b8-9a45-52486c4087a2/volumes"
Jul 13 23:46:50.614839 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.614808    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"]
Jul 13 23:46:50.615234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.615120    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15cf04cb-d20c-46b8-9a45-52486c4087a2" containerName="manager"
Jul 13 23:46:50.615234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.615131    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="15cf04cb-d20c-46b8-9a45-52486c4087a2" containerName="manager"
Jul 13 23:46:50.615234 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.615186    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="15cf04cb-d20c-46b8-9a45-52486c4087a2" containerName="manager"
Jul 13 23:46:50.636004 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.635975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"]
Jul 13 23:46:50.636177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.636109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.638600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.638581    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nvr8b\""
Jul 13 23:46:50.727853 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxc72\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-kube-api-access-sxc72\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727870    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.727993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728036 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.728033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.728052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.728261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.728074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829335 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829307    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829505 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829549    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxc72\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-kube-api-access-sxc72\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.829984 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.830023 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.829998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.830057 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.830023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.830396 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.830370    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.831619 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.831597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.831748 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.831730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.837837 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.837807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxc72\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-kube-api-access-sxc72\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.838025 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.838007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba0ef99-657e-4d31-bbfc-f0de7d28af9a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jtwnl\" (UID: \"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:50.948353 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:50.948271    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:51.082495 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:51.082464    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"]
Jul 13 23:46:51.085871 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:51.085845    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba0ef99_657e_4d31_bbfc_f0de7d28af9a.slice/crio-7df5dc41e673550c00f1e22cf59abbe7668779dc31d3ce7396bc705d8e4dc3c9 WatchSource:0}: Error finding container 7df5dc41e673550c00f1e22cf59abbe7668779dc31d3ce7396bc705d8e4dc3c9: Status 404 returned error can't find the container with id 7df5dc41e673550c00f1e22cf59abbe7668779dc31d3ce7396bc705d8e4dc3c9
Jul 13 23:46:51.267006 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:51.266973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" event={"ID":"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a","Type":"ContainerStarted","Data":"7df5dc41e673550c00f1e22cf59abbe7668779dc31d3ce7396bc705d8e4dc3c9"}
Jul 13 23:46:53.847901 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:53.847850    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:46:53.848221 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:53.847947    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:46:53.848221 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:53.847978    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:46:54.284565 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:54.284533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" event={"ID":"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a","Type":"ContainerStarted","Data":"77db23d1d2fb727bae6ed88fdf8c21e59340d972c5e1969e4fbb29f79b1337a0"}
Jul 13 23:46:54.307816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:54.307768    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" podStartSLOduration=1.548031408 podStartE2EDuration="4.307751295s" totalImagesPullingTime="2.759719887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:54.30704633 +0000 UTC m=+462.594264679" watchObservedRunningTime="2026-07-13 23:46:54.307751295 +0000 UTC m=+462.594969640"
Jul 13 23:46:54.948496 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:54.948463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:54.953282 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:54.953260    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:55.290016 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.289975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:55.291013 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.290996    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:46:55.369939 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.369908    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:46:55.399211 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.399189    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:46:55.399327 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.399275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.402059 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.402039    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 13 23:46:55.466421 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.466391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:46:55.573369 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.573265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.573531 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.573376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph77f\" (UniqueName: \"kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.674493 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.674448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph77f\" (UniqueName: \"kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.674659 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.674530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.675198 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.675180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.683187 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.683167    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph77f\" (UniqueName: \"kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f\") pod \"limitador-limitador-64c94fd64f-swfv5\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.709103 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.709081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:55.840160 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:55.840134    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:46:55.841818 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:46:55.841782    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44a16b1_1e19_4bcc_ac96_b07be6590998.slice/crio-585465f1d8a2d69ce5163ebaef94b0452497f6993224a4beec2934b86d4f42e2 WatchSource:0}: Error finding container 585465f1d8a2d69ce5163ebaef94b0452497f6993224a4beec2934b86d4f42e2: Status 404 returned error can't find the container with id 585465f1d8a2d69ce5163ebaef94b0452497f6993224a4beec2934b86d4f42e2
Jul 13 23:46:56.300296 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:56.300262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" event={"ID":"c44a16b1-1e19-4bcc-ac96-b07be6590998","Type":"ContainerStarted","Data":"585465f1d8a2d69ce5163ebaef94b0452497f6993224a4beec2934b86d4f42e2"}
Jul 13 23:46:59.315957 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:59.315921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" event={"ID":"c44a16b1-1e19-4bcc-ac96-b07be6590998","Type":"ContainerStarted","Data":"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"}
Jul 13 23:46:59.316348 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:59.316031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:46:59.335555 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:46:59.335510    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" podStartSLOduration=1.575566973 podStartE2EDuration="4.335497823s" totalImagesPullingTime="2.75993085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:46:59.333772204 +0000 UTC m=+467.620990550" watchObservedRunningTime="2026-07-13 23:46:59.335497823 +0000 UTC m=+467.622716169"
Jul 13 23:47:10.321354 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:10.321324    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:47:11.964595 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:11.964558    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:47:11.965026 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:11.964745    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" podUID="c44a16b1-1e19-4bcc-ac96-b07be6590998" containerName="limitador" containerID="cri-o://4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88" gracePeriod=30
Jul 13 23:47:12.502456 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.502434    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:47:12.601546 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.601490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph77f\" (UniqueName: \"kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f\") pod \"c44a16b1-1e19-4bcc-ac96-b07be6590998\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") "
Jul 13 23:47:12.601546 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.601528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file\") pod \"c44a16b1-1e19-4bcc-ac96-b07be6590998\" (UID: \"c44a16b1-1e19-4bcc-ac96-b07be6590998\") "
Jul 13 23:47:12.601879 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.601855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file" (OuterVolumeSpecName: "config-file") pod "c44a16b1-1e19-4bcc-ac96-b07be6590998" (UID: "c44a16b1-1e19-4bcc-ac96-b07be6590998"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 23:47:12.603475 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.603455    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f" (OuterVolumeSpecName: "kube-api-access-ph77f") pod "c44a16b1-1e19-4bcc-ac96-b07be6590998" (UID: "c44a16b1-1e19-4bcc-ac96-b07be6590998"). InnerVolumeSpecName "kube-api-access-ph77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:47:12.702337 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.702306    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph77f\" (UniqueName: \"kubernetes.io/projected/c44a16b1-1e19-4bcc-ac96-b07be6590998-kube-api-access-ph77f\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:47:12.702337 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:12.702332    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c44a16b1-1e19-4bcc-ac96-b07be6590998-config-file\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:47:13.377028 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.376993    2558 generic.go:356] "Generic (PLEG): container finished" podID="c44a16b1-1e19-4bcc-ac96-b07be6590998" containerID="4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88" exitCode=0
Jul 13 23:47:13.377474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.377055    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" event={"ID":"c44a16b1-1e19-4bcc-ac96-b07be6590998","Type":"ContainerDied","Data":"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"}
Jul 13 23:47:13.377474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.377072    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5"
Jul 13 23:47:13.377474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.377093    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-swfv5" event={"ID":"c44a16b1-1e19-4bcc-ac96-b07be6590998","Type":"ContainerDied","Data":"585465f1d8a2d69ce5163ebaef94b0452497f6993224a4beec2934b86d4f42e2"}
Jul 13 23:47:13.377474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.377114    2558 scope.go:117] "RemoveContainer" containerID="4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"
Jul 13 23:47:13.385291 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.385276    2558 scope.go:117] "RemoveContainer" containerID="4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"
Jul 13 23:47:13.385526 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:47:13.385509    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88\": container with ID starting with 4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88 not found: ID does not exist" containerID="4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"
Jul 13 23:47:13.385571 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.385533    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88"} err="failed to get container status \"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88\": rpc error: code = NotFound desc = could not find container \"4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88\": container with ID starting with 4e16e6eca9c0ee1ef63bc8d7bccf86530fe03687faf52425624fa4ac8f4b1e88 not found: ID does not exist"
Jul 13 23:47:13.397212 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.397190    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:47:13.401961 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.401939    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-swfv5"]
Jul 13 23:47:13.432086 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:13.432066    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44a16b1-1e19-4bcc-ac96-b07be6590998" path="/var/lib/kubelet/pods/c44a16b1-1e19-4bcc-ac96-b07be6590998/volumes"
Jul 13 23:47:16.975705 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.975664    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"]
Jul 13 23:47:16.976196 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.976178    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44a16b1-1e19-4bcc-ac96-b07be6590998" containerName="limitador"
Jul 13 23:47:16.976240 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.976200    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44a16b1-1e19-4bcc-ac96-b07be6590998" containerName="limitador"
Jul 13 23:47:16.976299 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.976289    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44a16b1-1e19-4bcc-ac96-b07be6590998" containerName="limitador"
Jul 13 23:47:16.979415 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.979391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:16.982509 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.982486    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 23:47:16.982630 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.982507    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 13 23:47:16.983232 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.983211    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 23:47:16.983313 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.983248    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hrmnt\""
Jul 13 23:47:16.987041 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:16.987019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"]
Jul 13 23:47:17.034222 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.034197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45bh\" (UniqueName: \"kubernetes.io/projected/9aa87a00-f087-4f38-8b36-bcdbf678d293-kube-api-access-m45bh\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.034357 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.034235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9aa87a00-f087-4f38-8b36-bcdbf678d293-data\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.135653 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.135619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45bh\" (UniqueName: \"kubernetes.io/projected/9aa87a00-f087-4f38-8b36-bcdbf678d293-kube-api-access-m45bh\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.135813 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.135660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9aa87a00-f087-4f38-8b36-bcdbf678d293-data\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.136034 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.136017    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9aa87a00-f087-4f38-8b36-bcdbf678d293-data\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.152722 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.152691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45bh\" (UniqueName: \"kubernetes.io/projected/9aa87a00-f087-4f38-8b36-bcdbf678d293-kube-api-access-m45bh\") pod \"postgres-69fb4594cf-4nkr5\" (UID: \"9aa87a00-f087-4f38-8b36-bcdbf678d293\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.292021 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.291992    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:17.423553 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:17.423500    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"]
Jul 13 23:47:17.425966 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:47:17.425941    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa87a00_f087_4f38_8b36_bcdbf678d293.slice/crio-48eff85a14bdbbdff8e6f8dd4a7ae33194dbacf5fe66d19553abd2239ce3ea62 WatchSource:0}: Error finding container 48eff85a14bdbbdff8e6f8dd4a7ae33194dbacf5fe66d19553abd2239ce3ea62: Status 404 returned error can't find the container with id 48eff85a14bdbbdff8e6f8dd4a7ae33194dbacf5fe66d19553abd2239ce3ea62
Jul 13 23:47:18.406600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:18.405932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5" event={"ID":"9aa87a00-f087-4f38-8b36-bcdbf678d293","Type":"ContainerStarted","Data":"48eff85a14bdbbdff8e6f8dd4a7ae33194dbacf5fe66d19553abd2239ce3ea62"}
Jul 13 23:47:22.427854 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:22.427771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5" event={"ID":"9aa87a00-f087-4f38-8b36-bcdbf678d293","Type":"ContainerStarted","Data":"0cd71a461faf84d08df9357d2af02600ce6ca9139381a749eeec3df188afc700"}
Jul 13 23:47:22.427854 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:22.427838    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:22.445348 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:22.445304    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5" podStartSLOduration=1.791798977 podStartE2EDuration="6.445290467s" totalImagesPullingTime="4.65349149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:47:22.443329477 +0000 UTC m=+490.730547825" watchObservedRunningTime="2026-07-13 23:47:22.445290467 +0000 UTC m=+490.732508814"
Jul 13 23:47:28.461850 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:28.461821    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4nkr5"
Jul 13 23:47:35.794260 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.794221    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:35.797694 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.797673    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:35.803090 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.803064    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 23:47:35.803232 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.803120    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 23:47:35.803232 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.803209    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-58jx6\""
Jul 13 23:47:35.811913 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.811876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:35.896525 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.896491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:35.896672 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.896548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5xp\" (UniqueName: \"kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:35.976316 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.976286    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:47:35.979538 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.979518    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:35.990158 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.990133    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:47:35.997426 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.997405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:35.997519 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.997460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5xp\" (UniqueName: \"kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:35.999678 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:35.999658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:36.005619 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.005598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5xp\" (UniqueName: \"kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp\") pod \"maas-controller-5d95495fc5-s7hv4\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") " pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:36.098828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.098747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsklm\" (UniqueName: \"kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.098828 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.098789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.107986 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.107965    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:36.199571 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.199539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsklm\" (UniqueName: \"kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.199732 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.199589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.202378 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.202353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.209935 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.209874    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsklm\" (UniqueName: \"kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm\") pod \"maas-controller-5c958c8fd5-jb9f5\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") " pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.235711 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.235620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:36.238223 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:47:36.238202    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73c3404_02ae_43fa_b5d6_af931f2d4c51.slice/crio-77a6de6df284c4995e999a92f447d67bf7cc55cd3788a19a097ed23e02b3a51b WatchSource:0}: Error finding container 77a6de6df284c4995e999a92f447d67bf7cc55cd3788a19a097ed23e02b3a51b: Status 404 returned error can't find the container with id 77a6de6df284c4995e999a92f447d67bf7cc55cd3788a19a097ed23e02b3a51b
Jul 13 23:47:36.289650 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.289625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:36.439689 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.439663    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:47:36.441527 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:47:36.441505    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05357d48_9aac_421b_96ce_67cf60d255d7.slice/crio-207f11b7d9f8cbb727d2de0416f8feb63bcb1e2f9118f737a1a6193188f21415 WatchSource:0}: Error finding container 207f11b7d9f8cbb727d2de0416f8feb63bcb1e2f9118f737a1a6193188f21415: Status 404 returned error can't find the container with id 207f11b7d9f8cbb727d2de0416f8feb63bcb1e2f9118f737a1a6193188f21415
Jul 13 23:47:36.483711 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.483679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" event={"ID":"05357d48-9aac-421b-96ce-67cf60d255d7","Type":"ContainerStarted","Data":"207f11b7d9f8cbb727d2de0416f8feb63bcb1e2f9118f737a1a6193188f21415"}
Jul 13 23:47:36.484718 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:36.484700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" event={"ID":"c73c3404-02ae-43fa-b5d6-af931f2d4c51","Type":"ContainerStarted","Data":"77a6de6df284c4995e999a92f447d67bf7cc55cd3788a19a097ed23e02b3a51b"}
Jul 13 23:47:39.505617 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.505511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" event={"ID":"05357d48-9aac-421b-96ce-67cf60d255d7","Type":"ContainerStarted","Data":"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"}
Jul 13 23:47:39.506112 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.505643    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:39.507542 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.507506    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" event={"ID":"c73c3404-02ae-43fa-b5d6-af931f2d4c51","Type":"ContainerStarted","Data":"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"}
Jul 13 23:47:39.507692 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.507648    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:39.538252 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.538188    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" podStartSLOduration=1.891323028 podStartE2EDuration="4.538168933s" totalImagesPullingTime="2.646845905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:47:39.536615511 +0000 UTC m=+507.823833870" watchObservedRunningTime="2026-07-13 23:47:39.538168933 +0000 UTC m=+507.825387280"
Jul 13 23:47:39.570831 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:39.570761    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" podStartSLOduration=1.730706745 podStartE2EDuration="4.570736528s" totalImagesPullingTime="2.840029783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:47:39.570350715 +0000 UTC m=+507.857569060" watchObservedRunningTime="2026-07-13 23:47:39.570736528 +0000 UTC m=+507.857954887"
Jul 13 23:47:50.514512 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.514480    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:50.514973 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.514577    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:47:50.573473 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.573444    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:50.573632 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.573611    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" podUID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" containerName="manager" containerID="cri-o://e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37" gracePeriod=30
Jul 13 23:47:50.817485 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.817464    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:50.920149 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.920124    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert\") pod \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") "
Jul 13 23:47:50.920301 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.920203    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb5xp\" (UniqueName: \"kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp\") pod \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\" (UID: \"c73c3404-02ae-43fa-b5d6-af931f2d4c51\") "
Jul 13 23:47:50.922257 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.922228    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c73c3404-02ae-43fa-b5d6-af931f2d4c51" (UID: "c73c3404-02ae-43fa-b5d6-af931f2d4c51"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:47:50.922257 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:50.922232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp" (OuterVolumeSpecName: "kube-api-access-qb5xp") pod "c73c3404-02ae-43fa-b5d6-af931f2d4c51" (UID: "c73c3404-02ae-43fa-b5d6-af931f2d4c51"). InnerVolumeSpecName "kube-api-access-qb5xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:47:51.021656 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.021628    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb5xp\" (UniqueName: \"kubernetes.io/projected/c73c3404-02ae-43fa-b5d6-af931f2d4c51-kube-api-access-qb5xp\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:47:51.021656 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.021652    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c73c3404-02ae-43fa-b5d6-af931f2d4c51-webhook-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:47:51.558981 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.558951    2558 generic.go:356] "Generic (PLEG): container finished" podID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" containerID="e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37" exitCode=0
Jul 13 23:47:51.559360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.558994    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d95495fc5-s7hv4"
Jul 13 23:47:51.559360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.559042    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" event={"ID":"c73c3404-02ae-43fa-b5d6-af931f2d4c51","Type":"ContainerDied","Data":"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"}
Jul 13 23:47:51.559360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.559089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d95495fc5-s7hv4" event={"ID":"c73c3404-02ae-43fa-b5d6-af931f2d4c51","Type":"ContainerDied","Data":"77a6de6df284c4995e999a92f447d67bf7cc55cd3788a19a097ed23e02b3a51b"}
Jul 13 23:47:51.559360 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.559117    2558 scope.go:117] "RemoveContainer" containerID="e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"
Jul 13 23:47:51.567101 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.567086    2558 scope.go:117] "RemoveContainer" containerID="e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"
Jul 13 23:47:51.567339 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:47:51.567321    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37\": container with ID starting with e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37 not found: ID does not exist" containerID="e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"
Jul 13 23:47:51.567386 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.567346    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37"} err="failed to get container status \"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37\": rpc error: code = NotFound desc = could not find container \"e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37\": container with ID starting with e82bd49071d99025056d6efa00ffb070cff6154fb10beefb5870127f0fb12e37 not found: ID does not exist"
Jul 13 23:47:51.575130 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.575105    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:51.579317 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:51.579293    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d95495fc5-s7hv4"]
Jul 13 23:47:53.431705 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:47:53.431673    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" path="/var/lib/kubelet/pods/c73c3404-02ae-43fa-b5d6-af931f2d4c51/volumes"
Jul 13 23:48:11.574253 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.574175    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"]
Jul 13 23:48:11.574610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.574531    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" containerName="manager"
Jul 13 23:48:11.574610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.574545    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" containerName="manager"
Jul 13 23:48:11.574610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.574606    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73c3404-02ae-43fa-b5d6-af931f2d4c51" containerName="manager"
Jul 13 23:48:11.578770 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.578754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.583345 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.583320    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 13 23:48:11.583482 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.583352    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9p8wm\""
Jul 13 23:48:11.590309 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.590283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"]
Jul 13 23:48:11.669460 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.669430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85adf18-0744-49e1-8b3d-9972351cfb6f-openshift-service-ca\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.669618 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.669466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85adf18-0744-49e1-8b3d-9972351cfb6f-maas-api-tls\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.669618 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.669530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246tb\" (UniqueName: \"kubernetes.io/projected/a85adf18-0744-49e1-8b3d-9972351cfb6f-kube-api-access-246tb\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.770671 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.770638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85adf18-0744-49e1-8b3d-9972351cfb6f-maas-api-tls\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.770818 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.770685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246tb\" (UniqueName: \"kubernetes.io/projected/a85adf18-0744-49e1-8b3d-9972351cfb6f-kube-api-access-246tb\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.770818 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.770745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85adf18-0744-49e1-8b3d-9972351cfb6f-openshift-service-ca\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.771370 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.771351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a85adf18-0744-49e1-8b3d-9972351cfb6f-openshift-service-ca\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.773060 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.773035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a85adf18-0744-49e1-8b3d-9972351cfb6f-maas-api-tls\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.784310 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.784287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246tb\" (UniqueName: \"kubernetes.io/projected/a85adf18-0744-49e1-8b3d-9972351cfb6f-kube-api-access-246tb\") pod \"maas-api-6dbd59cc95-nnlv7\" (UID: \"a85adf18-0744-49e1-8b3d-9972351cfb6f\") " pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:11.889226 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:11.889151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:12.020700 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.020674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"]
Jul 13 23:48:12.022908 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:48:12.022857    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85adf18_0744_49e1_8b3d_9972351cfb6f.slice/crio-1e46eee4bdacc182588f09f8b0e8f5dbde487bd8f91ad347f11ff2ebb5a08674 WatchSource:0}: Error finding container 1e46eee4bdacc182588f09f8b0e8f5dbde487bd8f91ad347f11ff2ebb5a08674: Status 404 returned error can't find the container with id 1e46eee4bdacc182588f09f8b0e8f5dbde487bd8f91ad347f11ff2ebb5a08674
Jul 13 23:48:12.199987 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.199908    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"]
Jul 13 23:48:12.203111 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.203095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.205613 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.205593    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 13 23:48:12.213943 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.213921    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"]
Jul 13 23:48:12.228566 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.228543    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"]
Jul 13 23:48:12.231029 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.231013    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.233630 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.233615    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hbtm2\""
Jul 13 23:48:12.242428 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.242407    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"]
Jul 13 23:48:12.274575 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.274547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa48b93-a93d-4ac4-837d-4e980f7232af-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.274686 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.274601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkczs\" (UniqueName: \"kubernetes.io/projected/5aa48b93-a93d-4ac4-837d-4e980f7232af-kube-api-access-dkczs\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.375173 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.375135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkczs\" (UniqueName: \"kubernetes.io/projected/5aa48b93-a93d-4ac4-837d-4e980f7232af-kube-api-access-dkczs\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.375317 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.375189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7121938-30a7-4af5-b35c-14a9e8ee0723-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.375317 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.375224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgt6\" (UniqueName: \"kubernetes.io/projected/f7121938-30a7-4af5-b35c-14a9e8ee0723-kube-api-access-nmgt6\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.375317 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.375277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa48b93-a93d-4ac4-837d-4e980f7232af-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.375916 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.375880    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aa48b93-a93d-4ac4-837d-4e980f7232af-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.383323 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.383301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkczs\" (UniqueName: \"kubernetes.io/projected/5aa48b93-a93d-4ac4-837d-4e980f7232af-kube-api-access-dkczs\") pod \"payload-pre-processing-74584bb9f8-8b64v\" (UID: \"5aa48b93-a93d-4ac4-837d-4e980f7232af\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.475613 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.475547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgt6\" (UniqueName: \"kubernetes.io/projected/f7121938-30a7-4af5-b35c-14a9e8ee0723-kube-api-access-nmgt6\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.475730 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.475619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7121938-30a7-4af5-b35c-14a9e8ee0723-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.476266 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.476245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7121938-30a7-4af5-b35c-14a9e8ee0723-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.483429 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.483397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgt6\" (UniqueName: \"kubernetes.io/projected/f7121938-30a7-4af5-b35c-14a9e8ee0723-kube-api-access-nmgt6\") pod \"payload-processing-f9b8d9b9b-5cj5s\" (UID: \"f7121938-30a7-4af5-b35c-14a9e8ee0723\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.514639 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.514616    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:12.540799 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.540774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:12.651947 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.651838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7" event={"ID":"a85adf18-0744-49e1-8b3d-9972351cfb6f","Type":"ContainerStarted","Data":"1e46eee4bdacc182588f09f8b0e8f5dbde487bd8f91ad347f11ff2ebb5a08674"}
Jul 13 23:48:12.655177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.655151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"]
Jul 13 23:48:12.657112 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:48:12.657075    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa48b93_a93d_4ac4_837d_4e980f7232af.slice/crio-969ddad3b7c50dfc9b162b0e7f8e50dd41a098541e67bf26c62f33f6b3cbe4e3 WatchSource:0}: Error finding container 969ddad3b7c50dfc9b162b0e7f8e50dd41a098541e67bf26c62f33f6b3cbe4e3: Status 404 returned error can't find the container with id 969ddad3b7c50dfc9b162b0e7f8e50dd41a098541e67bf26c62f33f6b3cbe4e3
Jul 13 23:48:12.687177 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:12.687152    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"]
Jul 13 23:48:12.690183 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:48:12.690153    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7121938_30a7_4af5_b35c_14a9e8ee0723.slice/crio-8726fe493e6a1c126ca4359f34ebe21c371a2a1a066682793723c970aa75bc07 WatchSource:0}: Error finding container 8726fe493e6a1c126ca4359f34ebe21c371a2a1a066682793723c970aa75bc07: Status 404 returned error can't find the container with id 8726fe493e6a1c126ca4359f34ebe21c371a2a1a066682793723c970aa75bc07
Jul 13 23:48:13.660997 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:13.660938    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s" event={"ID":"f7121938-30a7-4af5-b35c-14a9e8ee0723","Type":"ContainerStarted","Data":"8726fe493e6a1c126ca4359f34ebe21c371a2a1a066682793723c970aa75bc07"}
Jul 13 23:48:13.662589 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:13.662498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v" event={"ID":"5aa48b93-a93d-4ac4-837d-4e980f7232af","Type":"ContainerStarted","Data":"969ddad3b7c50dfc9b162b0e7f8e50dd41a098541e67bf26c62f33f6b3cbe4e3"}
Jul 13 23:48:14.668649 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.668618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v" event={"ID":"5aa48b93-a93d-4ac4-837d-4e980f7232af","Type":"ContainerStarted","Data":"66c5e76ad4169646ca7cb7d15c28463eb6f0e652044c643fea1d7ceba53c3d5c"}
Jul 13 23:48:14.669085 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.668854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:14.670056 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.670029    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7" event={"ID":"a85adf18-0744-49e1-8b3d-9972351cfb6f","Type":"ContainerStarted","Data":"b59f530458d3ddd92bc1b382bcc881502b594425c22f4f430b3939459a082e84"}
Jul 13 23:48:14.670457 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.670437    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:14.671772 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.671752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s" event={"ID":"f7121938-30a7-4af5-b35c-14a9e8ee0723","Type":"ContainerStarted","Data":"69903f5d162d207b2e73eef1d3794ef36910a54fa6cc469d406d3de42d97e38a"}
Jul 13 23:48:14.672147 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.672133    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:14.686059 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.686022    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v" podStartSLOduration=0.840950589 podStartE2EDuration="2.68600943s" totalImagesPullingTime="1.845058841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:48:14.684343594 +0000 UTC m=+542.971561941" watchObservedRunningTime="2026-07-13 23:48:14.68600943 +0000 UTC m=+542.973227770"
Jul 13 23:48:14.701137 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.701103    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s" podStartSLOduration=0.8864664 podStartE2EDuration="2.701092763s" totalImagesPullingTime="1.814626363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:48:14.699590471 +0000 UTC m=+542.986808818" watchObservedRunningTime="2026-07-13 23:48:14.701092763 +0000 UTC m=+542.988311108"
Jul 13 23:48:14.717268 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:14.717229    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7" podStartSLOduration=1.237524743 podStartE2EDuration="3.717216525s" totalImagesPullingTime="2.479691782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:48:14.716344168 +0000 UTC m=+543.003562512" watchObservedRunningTime="2026-07-13 23:48:14.717216525 +0000 UTC m=+543.004434937"
Jul 13 23:48:19.695654 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.695623    2558 generic.go:356] "Generic (PLEG): container finished" podID="3ba0ef99-657e-4d31-bbfc-f0de7d28af9a" containerID="77db23d1d2fb727bae6ed88fdf8c21e59340d972c5e1969e4fbb29f79b1337a0" exitCode=0
Jul 13 23:48:19.696137 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.695676    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" event={"ID":"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a","Type":"ContainerDied","Data":"77db23d1d2fb727bae6ed88fdf8c21e59340d972c5e1969e4fbb29f79b1337a0"}
Jul 13 23:48:19.696203 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.696157    2558 scope.go:117] "RemoveContainer" containerID="77db23d1d2fb727bae6ed88fdf8c21e59340d972c5e1969e4fbb29f79b1337a0"
Jul 13 23:48:19.697152 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.697125    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:48:19.697222 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.697193    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:48:19.697265 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:19.697224    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 13 23:48:20.702016 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:20.701978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" event={"ID":"3ba0ef99-657e-4d31-bbfc-f0de7d28af9a","Type":"ContainerStarted","Data":"f63f998aa3f97f746958e2eb804f30ba58719e7a0dbb7b77955143c317a72dc5"}
Jul 13 23:48:20.949410 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:20.949369    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:48:20.950602 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:20.950578    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jtwnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 13 23:48:20.950712 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:20.950634    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" podUID="3ba0ef99-657e-4d31-bbfc-f0de7d28af9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 13 23:48:21.686192 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:21.686164    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dbd59cc95-nnlv7"
Jul 13 23:48:21.949249 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:21.949153    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jtwnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 13 23:48:21.949249 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:21.949212    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" podUID="3ba0ef99-657e-4d31-bbfc-f0de7d28af9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 13 23:48:22.949673 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:22.949631    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jtwnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 13 23:48:22.950062 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:22.949692    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" podUID="3ba0ef99-657e-4d31-bbfc-f0de7d28af9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 13 23:48:23.949079 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:23.949046    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jtwnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 13 23:48:23.949258 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:23.949105    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl" podUID="3ba0ef99-657e-4d31-bbfc-f0de7d28af9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 13 23:48:24.952443 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:24.952412    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:48:24.952857 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:24.952688    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:48:24.953304 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:24.953286    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jtwnl"
Jul 13 23:48:25.679060 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:25.679033    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-8b64v"
Jul 13 23:48:26.347479 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.347447    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"]
Jul 13 23:48:26.351837 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.351816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.354581 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.354553    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 13 23:48:26.354581 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.354575    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fjg7x\""
Jul 13 23:48:26.354741 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.354695    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 13 23:48:26.355528 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.355511    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 13 23:48:26.362221 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.362198    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"]
Jul 13 23:48:26.387693 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387671    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.387789 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.387789 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.387789 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81f08ec-9427-44d9-b226-fd5df0291af0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.387917 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdh7\" (UniqueName: \"kubernetes.io/projected/a81f08ec-9427-44d9-b226-fd5df0291af0-kube-api-access-nvdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.387958 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.387917    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488415 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488590 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488590 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488590 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81f08ec-9427-44d9-b226-fd5df0291af0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488590 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdh7\" (UniqueName: \"kubernetes.io/projected/a81f08ec-9427-44d9-b226-fd5df0291af0-kube-api-access-nvdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488590 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488854 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.488854 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.489007 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.488906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.490776 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.490739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81f08ec-9427-44d9-b226-fd5df0291af0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.490935 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.490918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81f08ec-9427-44d9-b226-fd5df0291af0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.496932 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.496911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdh7\" (UniqueName: \"kubernetes.io/projected/a81f08ec-9427-44d9-b226-fd5df0291af0-kube-api-access-nvdh7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz\" (UID: \"a81f08ec-9427-44d9-b226-fd5df0291af0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.662459 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.662375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:26.683074 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.683051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-5cj5s"
Jul 13 23:48:26.807022 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:26.806997    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"]
Jul 13 23:48:26.808610 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:48:26.808569    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81f08ec_9427_44d9_b226_fd5df0291af0.slice/crio-6bfbe7d4bf2c8d168ce9f55361fe42954a483f546edc96365fe42f82ba69e49f WatchSource:0}: Error finding container 6bfbe7d4bf2c8d168ce9f55361fe42954a483f546edc96365fe42f82ba69e49f: Status 404 returned error can't find the container with id 6bfbe7d4bf2c8d168ce9f55361fe42954a483f546edc96365fe42f82ba69e49f
Jul 13 23:48:27.738650 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:27.738615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz" event={"ID":"a81f08ec-9427-44d9-b226-fd5df0291af0","Type":"ContainerStarted","Data":"6bfbe7d4bf2c8d168ce9f55361fe42954a483f546edc96365fe42f82ba69e49f"}
Jul 13 23:48:28.749256 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:28.749220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz" event={"ID":"a81f08ec-9427-44d9-b226-fd5df0291af0","Type":"ContainerStarted","Data":"7dac44cf55d6687050fa26a09a53c4ebd6afbb7b6f803ba6bf3f716467db4304"}
Jul 13 23:48:28.767443 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:28.767395    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz" podStartSLOduration=1.604705006 podStartE2EDuration="2.767380578s" totalImagesPullingTime="1.162675572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:48:28.765850455 +0000 UTC m=+557.053068801" watchObservedRunningTime="2026-07-13 23:48:28.767380578 +0000 UTC m=+557.054598924"
Jul 13 23:48:36.662497 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:36.662461    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:36.662875 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:36.662515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:36.674749 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:36.674725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:36.795242 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:36.795207    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz"
Jul 13 23:48:47.736713 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.736672    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"]
Jul 13 23:48:47.742308 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.742284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.744763 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.744745    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 13 23:48:47.750448 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.750429    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"]
Jul 13 23:48:47.866548 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.866548 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.866755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866575    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.866755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn9xd\" (UniqueName: \"kubernetes.io/projected/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-kube-api-access-cn9xd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.866755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.866755 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.866680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968078 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968043    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968078 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968289 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968289 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn9xd\" (UniqueName: \"kubernetes.io/projected/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-kube-api-access-cn9xd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968289 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968289 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968642 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968642 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.968811 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.968680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.970351 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.970325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.970594 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.970577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:47.979715 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:47.979693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn9xd\" (UniqueName: \"kubernetes.io/projected/db4872f5-1fa4-4c92-b144-065d2ab3fdaa-kube-api-access-cn9xd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-z6mbh\" (UID: \"db4872f5-1fa4-4c92-b144-065d2ab3fdaa\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:48.053233 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:48.053205    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:48.183509 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:48.183483    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"]
Jul 13 23:48:48.185161 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:48:48.185131    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4872f5_1fa4_4c92_b144_065d2ab3fdaa.slice/crio-5d74c046c6785770cd0342f5c6249b75b0c2a946cbfa27076a9220936bf50abf WatchSource:0}: Error finding container 5d74c046c6785770cd0342f5c6249b75b0c2a946cbfa27076a9220936bf50abf: Status 404 returned error can't find the container with id 5d74c046c6785770cd0342f5c6249b75b0c2a946cbfa27076a9220936bf50abf
Jul 13 23:48:48.836816 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:48.836780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh" event={"ID":"db4872f5-1fa4-4c92-b144-065d2ab3fdaa","Type":"ContainerStarted","Data":"94edb74fc08caac485f8b3af3d6b4913bdd1f5fe6e22541460a711d11559a556"}
Jul 13 23:48:48.837201 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:48.836847    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh" event={"ID":"db4872f5-1fa4-4c92-b144-065d2ab3fdaa","Type":"ContainerStarted","Data":"5d74c046c6785770cd0342f5c6249b75b0c2a946cbfa27076a9220936bf50abf"}
Jul 13 23:48:48.855672 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:48.855631    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh" podStartSLOduration=1.6939838790000001 podStartE2EDuration="1.855616577s" totalImagesPullingTime="161.632698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:48:48.853248693 +0000 UTC m=+577.140467036" watchObservedRunningTime="2026-07-13 23:48:48.855616577 +0000 UTC m=+577.142834924"
Jul 13 23:48:58.053564 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:58.053521    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:58.054025 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:58.053597    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:58.067121 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:58.067095    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:48:58.891521 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:48:58.891495    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-z6mbh"
Jul 13 23:51:34.674430 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:34.674332    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:51:34.674986 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:34.674813    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" podUID="05357d48-9aac-421b-96ce-67cf60d255d7" containerName="manager" containerID="cri-o://84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384" gracePeriod=30
Jul 13 23:51:34.920873 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:34.920850    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:51:35.077133 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.077103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsklm\" (UniqueName: \"kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm\") pod \"05357d48-9aac-421b-96ce-67cf60d255d7\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") "
Jul 13 23:51:35.077285 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.077222    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert\") pod \"05357d48-9aac-421b-96ce-67cf60d255d7\" (UID: \"05357d48-9aac-421b-96ce-67cf60d255d7\") "
Jul 13 23:51:35.079053 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.079023    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm" (OuterVolumeSpecName: "kube-api-access-zsklm") pod "05357d48-9aac-421b-96ce-67cf60d255d7" (UID: "05357d48-9aac-421b-96ce-67cf60d255d7"). InnerVolumeSpecName "kube-api-access-zsklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:51:35.079158 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.079103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "05357d48-9aac-421b-96ce-67cf60d255d7" (UID: "05357d48-9aac-421b-96ce-67cf60d255d7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 23:51:35.178668 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.178639    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05357d48-9aac-421b-96ce-67cf60d255d7-webhook-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:51:35.178668 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.178664    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsklm\" (UniqueName: \"kubernetes.io/projected/05357d48-9aac-421b-96ce-67cf60d255d7-kube-api-access-zsklm\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:51:35.587437 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.587403    2558 generic.go:356] "Generic (PLEG): container finished" podID="05357d48-9aac-421b-96ce-67cf60d255d7" containerID="84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384" exitCode=0
Jul 13 23:51:35.587600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.587448    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5"
Jul 13 23:51:35.587600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.587492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" event={"ID":"05357d48-9aac-421b-96ce-67cf60d255d7","Type":"ContainerDied","Data":"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"}
Jul 13 23:51:35.587600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.587540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-jb9f5" event={"ID":"05357d48-9aac-421b-96ce-67cf60d255d7","Type":"ContainerDied","Data":"207f11b7d9f8cbb727d2de0416f8feb63bcb1e2f9118f737a1a6193188f21415"}
Jul 13 23:51:35.587600 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.587561    2558 scope.go:117] "RemoveContainer" containerID="84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"
Jul 13 23:51:35.595812 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.595801    2558 scope.go:117] "RemoveContainer" containerID="84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"
Jul 13 23:51:35.596058 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:51:35.596041    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384\": container with ID starting with 84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384 not found: ID does not exist" containerID="84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"
Jul 13 23:51:35.596122 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.596067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384"} err="failed to get container status \"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384\": rpc error: code = NotFound desc = could not find container \"84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384\": container with ID starting with 84224479e312ebe6c006ad831a5e5ae8a27e8383bc48f2a6bd61fe2ac564f384 not found: ID does not exist"
Jul 13 23:51:35.605519 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.605497    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:51:35.611737 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:35.611717    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-jb9f5"]
Jul 13 23:51:36.369261 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.369211    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-6hcz7"]
Jul 13 23:51:36.369735 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.369714    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05357d48-9aac-421b-96ce-67cf60d255d7" containerName="manager"
Jul 13 23:51:36.369810 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.369738    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="05357d48-9aac-421b-96ce-67cf60d255d7" containerName="manager"
Jul 13 23:51:36.369862 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.369854    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="05357d48-9aac-421b-96ce-67cf60d255d7" containerName="manager"
Jul 13 23:51:36.374179 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.374157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.376791 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.376775    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 13 23:51:36.377754 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.377741    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-58jx6\""
Jul 13 23:51:36.377823 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.377749    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 13 23:51:36.383068 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.383045    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-6hcz7"]
Jul 13 23:51:36.489922 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.489872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/693d369c-d984-4b3a-8ba1-1f1241b382d1-webhook-cert\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.490078 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.489934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2gx\" (UniqueName: \"kubernetes.io/projected/693d369c-d984-4b3a-8ba1-1f1241b382d1-kube-api-access-np2gx\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.591118 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.591087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/693d369c-d984-4b3a-8ba1-1f1241b382d1-webhook-cert\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.591298 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.591133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2gx\" (UniqueName: \"kubernetes.io/projected/693d369c-d984-4b3a-8ba1-1f1241b382d1-kube-api-access-np2gx\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.593610 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.593582    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/693d369c-d984-4b3a-8ba1-1f1241b382d1-webhook-cert\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.599021 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.599000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2gx\" (UniqueName: \"kubernetes.io/projected/693d369c-d984-4b3a-8ba1-1f1241b382d1-kube-api-access-np2gx\") pod \"maas-controller-5c958c8fd5-6hcz7\" (UID: \"693d369c-d984-4b3a-8ba1-1f1241b382d1\") " pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.684742 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.684678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:36.814933 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.814843    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c958c8fd5-6hcz7"]
Jul 13 23:51:36.817219 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:51:36.817190    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693d369c_d984_4b3a_8ba1_1f1241b382d1.slice/crio-c63d49b71a1f330b4db66d55d4e49fec15c3e2e3e03856a3413f393135401b97 WatchSource:0}: Error finding container c63d49b71a1f330b4db66d55d4e49fec15c3e2e3e03856a3413f393135401b97: Status 404 returned error can't find the container with id c63d49b71a1f330b4db66d55d4e49fec15c3e2e3e03856a3413f393135401b97
Jul 13 23:51:36.818452 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:36.818435    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 23:51:37.433297 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:37.433269    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05357d48-9aac-421b-96ce-67cf60d255d7" path="/var/lib/kubelet/pods/05357d48-9aac-421b-96ce-67cf60d255d7/volumes"
Jul 13 23:51:37.599268 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:37.599234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7" event={"ID":"693d369c-d984-4b3a-8ba1-1f1241b382d1","Type":"ContainerStarted","Data":"adebe761b4898cb53b7c08f8126794442c444cddae9fcd179a0060f6a0e50a1d"}
Jul 13 23:51:37.599431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:37.599285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7" event={"ID":"693d369c-d984-4b3a-8ba1-1f1241b382d1","Type":"ContainerStarted","Data":"c63d49b71a1f330b4db66d55d4e49fec15c3e2e3e03856a3413f393135401b97"}
Jul 13 23:51:37.599431 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:37.599377    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:51:37.616287 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:37.616243    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7" podStartSLOduration=1.053283692 podStartE2EDuration="1.616229621s" totalImagesPullingTime="562.945929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:51:37.615013438 +0000 UTC m=+745.902231796" watchObservedRunningTime="2026-07-13 23:51:37.616229621 +0000 UTC m=+745.903447967"
Jul 13 23:51:48.606476 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:51:48.606449    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c958c8fd5-6hcz7"
Jul 13 23:56:25.950824 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:25.950789    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:56:25.951399 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:25.951030    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" podUID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" containerName="manager" containerID="cri-o://07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52" gracePeriod=10
Jul 13 23:56:29.304759 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.304730    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:56:29.457087 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.456998    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrvg\" (UniqueName: \"kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg\") pod \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") "
Jul 13 23:56:29.457087 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.457075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume\") pod \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\" (UID: \"9ffc6c7f-0e02-407e-a28c-64087b0ed06e\") "
Jul 13 23:56:29.457448 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.457422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9ffc6c7f-0e02-407e-a28c-64087b0ed06e" (UID: "9ffc6c7f-0e02-407e-a28c-64087b0ed06e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 23:56:29.459213 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.459191    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg" (OuterVolumeSpecName: "kube-api-access-mzrvg") pod "9ffc6c7f-0e02-407e-a28c-64087b0ed06e" (UID: "9ffc6c7f-0e02-407e-a28c-64087b0ed06e"). InnerVolumeSpecName "kube-api-access-mzrvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 23:56:29.558385 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.558357    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrvg\" (UniqueName: \"kubernetes.io/projected/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-kube-api-access-mzrvg\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:56:29.558385 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.558381    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ffc6c7f-0e02-407e-a28c-64087b0ed06e-extensions-socket-volume\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 13 23:56:29.932226 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.932190    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" containerID="07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52" exitCode=0
Jul 13 23:56:29.932394 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.932238    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"
Jul 13 23:56:29.932394 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.932247    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" event={"ID":"9ffc6c7f-0e02-407e-a28c-64087b0ed06e","Type":"ContainerDied","Data":"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"}
Jul 13 23:56:29.932394 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.932297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz" event={"ID":"9ffc6c7f-0e02-407e-a28c-64087b0ed06e","Type":"ContainerDied","Data":"93268a21a0a105dc910db686be47397187d212fae8767128cc1878ceec9dce2a"}
Jul 13 23:56:29.932394 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.932321    2558 scope.go:117] "RemoveContainer" containerID="07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"
Jul 13 23:56:29.942268 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.942239    2558 scope.go:117] "RemoveContainer" containerID="07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"
Jul 13 23:56:29.942516 ip-10-0-128-174 kubenswrapper[2558]: E0713 23:56:29.942496    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52\": container with ID starting with 07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52 not found: ID does not exist" containerID="07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"
Jul 13 23:56:29.942584 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.942525    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52"} err="failed to get container status \"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52\": rpc error: code = NotFound desc = could not find container \"07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52\": container with ID starting with 07fb9867881fb69f14a32b56db746d8276592cf5b69f13f0a1b87badc6242d52 not found: ID does not exist"
Jul 13 23:56:29.953617 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.953592    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:56:29.957854 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:29.957834    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-78blz"]
Jul 13 23:56:31.432038 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:56:31.432006    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" path="/var/lib/kubelet/pods/9ffc6c7f-0e02-407e-a28c-64087b0ed06e/volumes"
Jul 13 23:57:27.104855 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.104820    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"]
Jul 13 23:57:27.105251 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.105230    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" containerName="manager"
Jul 13 23:57:27.105251 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.105242    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" containerName="manager"
Jul 13 23:57:27.105324 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.105290    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ffc6c7f-0e02-407e-a28c-64087b0ed06e" containerName="manager"
Jul 13 23:57:27.108207 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.108191    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.110809 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.110787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r24xl\""
Jul 13 23:57:27.119474 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.119454    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"]
Jul 13 23:57:27.183063 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.183030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzkz\" (UniqueName: \"kubernetes.io/projected/73ca59af-1127-4868-9fca-4096e86e2f4f-kube-api-access-vbzkz\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.183201 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.183092    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73ca59af-1127-4868-9fca-4096e86e2f4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.284202 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.284173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzkz\" (UniqueName: \"kubernetes.io/projected/73ca59af-1127-4868-9fca-4096e86e2f4f-kube-api-access-vbzkz\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.284328 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.284228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73ca59af-1127-4868-9fca-4096e86e2f4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.284555 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.284539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/73ca59af-1127-4868-9fca-4096e86e2f4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.298221 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.298194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzkz\" (UniqueName: \"kubernetes.io/projected/73ca59af-1127-4868-9fca-4096e86e2f4f-kube-api-access-vbzkz\") pod \"kuadrant-operator-controller-manager-86878c769f-8frvx\" (UID: \"73ca59af-1127-4868-9fca-4096e86e2f4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.419174 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.419120    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:27.564776 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:27.564755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"]
Jul 13 23:57:27.566947 ip-10-0-128-174 kubenswrapper[2558]: W0713 23:57:27.566916    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ca59af_1127_4868_9fca_4096e86e2f4f.slice/crio-b311463ed1b7055f0775b207b45d7a4f09b3cd2361d614274da540cb57812869 WatchSource:0}: Error finding container b311463ed1b7055f0775b207b45d7a4f09b3cd2361d614274da540cb57812869: Status 404 returned error can't find the container with id b311463ed1b7055f0775b207b45d7a4f09b3cd2361d614274da540cb57812869
Jul 13 23:57:28.188358 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:28.188322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx" event={"ID":"73ca59af-1127-4868-9fca-4096e86e2f4f","Type":"ContainerStarted","Data":"1f8ea81ed53afc40b9031ec60a8f5768fb1365603b734c495ded656f8f9ddb1b"}
Jul 13 23:57:28.188721 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:28.188378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx" event={"ID":"73ca59af-1127-4868-9fca-4096e86e2f4f","Type":"ContainerStarted","Data":"b311463ed1b7055f0775b207b45d7a4f09b3cd2361d614274da540cb57812869"}
Jul 13 23:57:28.188721 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:28.188416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 13 23:57:28.209744 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:28.209697    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx" podStartSLOduration=1.209682324 podStartE2EDuration="1.209682324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 23:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 23:57:28.207328297 +0000 UTC m=+1096.494546644" watchObservedRunningTime="2026-07-13 23:57:28.209682324 +0000 UTC m=+1096.496900669"
Jul 13 23:57:39.195200 ip-10-0-128-174 kubenswrapper[2558]: I0713 23:57:39.195173    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8frvx"
Jul 14 00:00:00.151403 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.151368    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"]
Jul 14 00:00:00.154576 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.154554    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29733120-x7prj"]
Jul 14 00:00:00.154695 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.154668    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"
Jul 14 00:00:00.157407 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.157385    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:00:00.157645 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.157629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.159963 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.159938    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Jul 14 00:00:00.160064 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.160047    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-b4qps\""
Jul 14 00:00:00.171774 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.171755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29733120-x7prj"]
Jul 14 00:00:00.174524 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.174504    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"]
Jul 14 00:00:00.196419 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.196393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"
Jul 14 00:00:00.289867 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.289586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wg88\" (UniqueName: \"kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.289867 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.289682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.334575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.334382    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"]
Jul 14 00:00:00.337515 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:00:00.337482    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc268ed84_8ec4_4aaf_8024_51472f5ff92e.slice/crio-798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d WatchSource:0}: Error finding container 798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d: Status 404 returned error can't find the container with id 798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d
Jul 14 00:00:00.339547 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.339526    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:00:00.390973 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.390939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.391123 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.391030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wg88\" (UniqueName: \"kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.391525 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.391505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.400331 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.400307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wg88\" (UniqueName: \"kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88\") pod \"image-pruner-29733120-x7prj\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") " pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.504410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.504310    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:00.636690 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.636660    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29733120-x7prj"]
Jul 14 00:00:00.638094 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:00:00.638070    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee18ed3_6769_456a_9a49_cff272f60555.slice/crio-7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68 WatchSource:0}: Error finding container 7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68: Status 404 returned error can't find the container with id 7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68
Jul 14 00:00:00.879767 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.879713    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29733120-x7prj" event={"ID":"9ee18ed3-6769-456a-9a49-cff272f60555","Type":"ContainerStarted","Data":"294f5968557a437dca16c7ccdb75a50369f650f495395a4ee2089ade56b9868a"}
Jul 14 00:00:00.880020 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.879781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29733120-x7prj" event={"ID":"9ee18ed3-6769-456a-9a49-cff272f60555","Type":"ContainerStarted","Data":"7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68"}
Jul 14 00:00:00.881024 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.881000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf" event={"ID":"c268ed84-8ec4-4aaf-8024-51472f5ff92e","Type":"ContainerStarted","Data":"798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d"}
Jul 14 00:00:00.901741 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:00.901675    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29733120-x7prj" podStartSLOduration=0.901652546 podStartE2EDuration="901.652546ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:00:00.899092969 +0000 UTC m=+1249.186311316" watchObservedRunningTime="2026-07-14 00:00:00.901652546 +0000 UTC m=+1249.188870894"
Jul 14 00:00:01.888011 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:01.887979    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ee18ed3-6769-456a-9a49-cff272f60555" containerID="294f5968557a437dca16c7ccdb75a50369f650f495395a4ee2089ade56b9868a" exitCode=0
Jul 14 00:00:01.888371 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:01.888063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29733120-x7prj" event={"ID":"9ee18ed3-6769-456a-9a49-cff272f60555","Type":"ContainerDied","Data":"294f5968557a437dca16c7ccdb75a50369f650f495395a4ee2089ade56b9868a"}
Jul 14 00:00:03.035649 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.035622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:03.115208 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.115174    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca\") pod \"9ee18ed3-6769-456a-9a49-cff272f60555\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") "
Jul 14 00:00:03.115400 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.115247    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wg88\" (UniqueName: \"kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88\") pod \"9ee18ed3-6769-456a-9a49-cff272f60555\" (UID: \"9ee18ed3-6769-456a-9a49-cff272f60555\") "
Jul 14 00:00:03.115575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.115553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca" (OuterVolumeSpecName: "serviceca") pod "9ee18ed3-6769-456a-9a49-cff272f60555" (UID: "9ee18ed3-6769-456a-9a49-cff272f60555"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:00:03.117347 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.117323    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88" (OuterVolumeSpecName: "kube-api-access-7wg88") pod "9ee18ed3-6769-456a-9a49-cff272f60555" (UID: "9ee18ed3-6769-456a-9a49-cff272f60555"). InnerVolumeSpecName "kube-api-access-7wg88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:00:03.216560 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.216483    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wg88\" (UniqueName: \"kubernetes.io/projected/9ee18ed3-6769-456a-9a49-cff272f60555-kube-api-access-7wg88\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:00:03.216560 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.216508    2558 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ee18ed3-6769-456a-9a49-cff272f60555-serviceca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:00:03.899053 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.899023    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29733120-x7prj"
Jul 14 00:00:03.899228 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.899025    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29733120-x7prj" event={"ID":"9ee18ed3-6769-456a-9a49-cff272f60555","Type":"ContainerDied","Data":"7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68"}
Jul 14 00:00:03.899228 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:03.899146    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f59441a406875fd7ef86507d1209ffa99f908f54f95d10f2f12c3ab4bf0ac68"
Jul 14 00:00:06.915843 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:06.915805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf" event={"ID":"c268ed84-8ec4-4aaf-8024-51472f5ff92e","Type":"ContainerStarted","Data":"b7f42cc81e4a7600a40adffcb5edd1759da2c7aded04c63106ea51b0cb6a90b5"}
Jul 14 00:00:06.932978 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:06.932920    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf" podStartSLOduration=0.850247146 podStartE2EDuration="6.932902286s" totalImagesPullingTime="6.08265514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:00:06.931307029 +0000 UTC m=+1255.218525376" watchObservedRunningTime="2026-07-14 00:00:06.932902286 +0000 UTC m=+1255.220120622"
Jul 14 00:00:07.921525 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:07.921433    2558 generic.go:356] "Generic (PLEG): container finished" podID="c268ed84-8ec4-4aaf-8024-51472f5ff92e" containerID="b7f42cc81e4a7600a40adffcb5edd1759da2c7aded04c63106ea51b0cb6a90b5" exitCode=0
Jul 14 00:00:07.921525 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:07.921488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf" event={"ID":"c268ed84-8ec4-4aaf-8024-51472f5ff92e","Type":"ContainerDied","Data":"b7f42cc81e4a7600a40adffcb5edd1759da2c7aded04c63106ea51b0cb6a90b5"}
Jul 14 00:00:09.075337 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:09.075312    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"
Jul 14 00:00:09.933553 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:09.933524    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf"
Jul 14 00:00:09.933744 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:09.933564    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733120-42kmf" event={"ID":"c268ed84-8ec4-4aaf-8024-51472f5ff92e","Type":"ContainerDied","Data":"798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d"}
Jul 14 00:00:09.933744 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:00:09.933610    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798dd4f7358e5e4c99bd5c68b6446fb89959102aa7184d7859e94ea79c4be54d"
Jul 14 00:03:49.654164 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654120    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654528    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee18ed3-6769-456a-9a49-cff272f60555" containerName="image-pruner"
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654540    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee18ed3-6769-456a-9a49-cff272f60555" containerName="image-pruner"
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654555    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c268ed84-8ec4-4aaf-8024-51472f5ff92e" containerName="cleanup"
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654561    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c268ed84-8ec4-4aaf-8024-51472f5ff92e" containerName="cleanup"
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654616    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee18ed3-6769-456a-9a49-cff272f60555" containerName="image-pruner"
Jul 14 00:03:49.654948 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.654627    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c268ed84-8ec4-4aaf-8024-51472f5ff92e" containerName="cleanup"
Jul 14 00:03:49.656660 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.656643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.659112 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.659096    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-30780878-openshift-default-dockercfg-krkwp\""
Jul 14 00:03:49.681506 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.681474    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:03:49.763149 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rqn\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763321 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763321 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763321 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763321 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.763623 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.763548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.864808 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rqn\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864876    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.864988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865610 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865669 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.865727 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.865695    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.867379 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.867357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.867580 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.867565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.872858 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.872833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rqn\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.872995 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.872954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token\") pod \"e2e-ait-30780878-openshift-default-6dc654f6b5-2684x\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") " pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:49.968073 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:49.968010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:50.100676 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.100652    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:03:50.102759 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:03:50.102732    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420130a8_38ef_458c_9e54_a56ab38e636c.slice/crio-e3b4fdf4b4e994f949481b3657250fc8fdbecdfb559a6027115e5aef1bae58e1 WatchSource:0}: Error finding container e3b4fdf4b4e994f949481b3657250fc8fdbecdfb559a6027115e5aef1bae58e1: Status 404 returned error can't find the container with id e3b4fdf4b4e994f949481b3657250fc8fdbecdfb559a6027115e5aef1bae58e1
Jul 14 00:03:50.105294 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.105263    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:03:50.105380 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.105329    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:03:50.105380 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.105363    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:03:50.962316 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.962274    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerStarted","Data":"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"}
Jul 14 00:03:50.962737 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.962330    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerStarted","Data":"e3b4fdf4b4e994f949481b3657250fc8fdbecdfb559a6027115e5aef1bae58e1"}
Jul 14 00:03:50.968579 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.968545    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:50.985117 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:50.985077    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podStartSLOduration=1.985063443 podStartE2EDuration="1.985063443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:03:50.982602177 +0000 UTC m=+1479.269820535" watchObservedRunningTime="2026-07-14 00:03:50.985063443 +0000 UTC m=+1479.272281857"
Jul 14 00:03:51.085448 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.085408    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:03:51.088354 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.088331    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.090961 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.090933    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-30780878-serving-cert\""
Jul 14 00:03:51.099568 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.099529    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:03:51.176908 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.176851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.177115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.177004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.177115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.177034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pc4b\" (UniqueName: \"kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.278078 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.278040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.278078 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.278078    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pc4b\" (UniqueName: \"kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.278276 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.278150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.278690 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.278673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.280542 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.280526    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.287363 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.287333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pc4b\" (UniqueName: \"kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b\") pod \"maas-api-e2e-ait-30780878-566bc887ff-rfp7l\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.402421 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.402377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:51.533221 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.533166    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:03:51.535920 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:03:51.535866    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dd826b_f709_4f30_8732_b473baa52021.slice/crio-603c179263c775d0a8332820570da6dc310288ff08a6278f66314b535362a830 WatchSource:0}: Error finding container 603c179263c775d0a8332820570da6dc310288ff08a6278f66314b535362a830: Status 404 returned error can't find the container with id 603c179263c775d0a8332820570da6dc310288ff08a6278f66314b535362a830
Jul 14 00:03:51.968556 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.968475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" event={"ID":"f4dd826b-f709-4f30-8732-b473baa52021","Type":"ContainerStarted","Data":"603c179263c775d0a8332820570da6dc310288ff08a6278f66314b535362a830"}
Jul 14 00:03:51.972954 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:51.972910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:52.974573 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:52.974536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" event={"ID":"f4dd826b-f709-4f30-8732-b473baa52021","Type":"ContainerStarted","Data":"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"}
Jul 14 00:03:52.975087 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:52.974760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:52.975087 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:52.974807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:03:52.975862 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:52.975847    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:03:52.994753 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:52.994696    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" podStartSLOduration=1.238501951 podStartE2EDuration="1.994676057s" totalImagesPullingTime="756.174106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:03:52.993960168 +0000 UTC m=+1481.281178515" watchObservedRunningTime="2026-07-14 00:03:52.994676057 +0000 UTC m=+1481.281894404"
Jul 14 00:03:58.985298 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:03:58.985260    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:04:03.024572 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.024546    2558 generic.go:356] "Generic (PLEG): container finished" podID="420130a8-38ef-458c-9e54-a56ab38e636c" containerID="6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d" exitCode=0
Jul 14 00:04:03.025008 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.024618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerDied","Data":"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"}
Jul 14 00:04:03.025129 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.025113    2558 scope.go:117] "RemoveContainer" containerID="6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"
Jul 14 00:04:03.026283 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.026247    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:04:03.026379 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.026352    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:04:03.026432 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:03.026415    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:04:04.035651 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:04.034685    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerStarted","Data":"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"}
Jul 14 00:04:04.969272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:04.969236    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:04.970141 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:04.970116    2558 patch_prober.go:28] interesting pod/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 14 00:04:04.970283 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:04.970169    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 14 00:04:05.969146 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:05.969098    2558 patch_prober.go:28] interesting pod/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 14 00:04:05.969521 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:05.969183    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 14 00:04:06.496264 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.496226    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:04:06.496532 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.496483    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" podUID="f4dd826b-f709-4f30-8732-b473baa52021" containerName="maas-api" containerID="cri-o://6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6" gracePeriod=30
Jul 14 00:04:06.800315 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.800293    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:04:06.950170 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.950126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls\") pod \"f4dd826b-f709-4f30-8732-b473baa52021\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") "
Jul 14 00:04:06.950170 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.950165    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca\") pod \"f4dd826b-f709-4f30-8732-b473baa52021\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") "
Jul 14 00:04:06.950428 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.950268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pc4b\" (UniqueName: \"kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b\") pod \"f4dd826b-f709-4f30-8732-b473baa52021\" (UID: \"f4dd826b-f709-4f30-8732-b473baa52021\") "
Jul 14 00:04:06.950635 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.950586    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4dd826b-f709-4f30-8732-b473baa52021" (UID: "f4dd826b-f709-4f30-8732-b473baa52021"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:04:06.952494 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.952469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b" (OuterVolumeSpecName: "kube-api-access-9pc4b") pod "f4dd826b-f709-4f30-8732-b473baa52021" (UID: "f4dd826b-f709-4f30-8732-b473baa52021"). InnerVolumeSpecName "kube-api-access-9pc4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:04:06.952599 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.952562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4dd826b-f709-4f30-8732-b473baa52021" (UID: "f4dd826b-f709-4f30-8732-b473baa52021"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:04:06.968966 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.968941    2558 patch_prober.go:28] interesting pod/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 14 00:04:06.969045 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:06.969000    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 14 00:04:07.051371 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.051251    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4dd826b-f709-4f30-8732-b473baa52021-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:07.051371 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.051281    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4dd826b-f709-4f30-8732-b473baa52021-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:07.051371 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.051295    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pc4b\" (UniqueName: \"kubernetes.io/projected/f4dd826b-f709-4f30-8732-b473baa52021-kube-api-access-9pc4b\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:07.055769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.055740    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4dd826b-f709-4f30-8732-b473baa52021" containerID="6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6" exitCode=0
Jul 14 00:04:07.055924 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.055787    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"
Jul 14 00:04:07.055924 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.055827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" event={"ID":"f4dd826b-f709-4f30-8732-b473baa52021","Type":"ContainerDied","Data":"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"}
Jul 14 00:04:07.055924 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.055880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l" event={"ID":"f4dd826b-f709-4f30-8732-b473baa52021","Type":"ContainerDied","Data":"603c179263c775d0a8332820570da6dc310288ff08a6278f66314b535362a830"}
Jul 14 00:04:07.056090 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.055937    2558 scope.go:117] "RemoveContainer" containerID="6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"
Jul 14 00:04:07.064594 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.064580    2558 scope.go:117] "RemoveContainer" containerID="6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"
Jul 14 00:04:07.064842 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:07.064816    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6\": container with ID starting with 6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6 not found: ID does not exist" containerID="6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"
Jul 14 00:04:07.064956 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.064849    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6"} err="failed to get container status \"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6\": rpc error: code = NotFound desc = could not find container \"6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6\": container with ID starting with 6d36c6558f91e0449cd9caaa43ddca3a06b6c98b4d94878dab4251da7cdae0d6 not found: ID does not exist"
Jul 14 00:04:07.077244 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.077224    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:04:07.079020 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.079003    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-30780878-566bc887ff-rfp7l"]
Jul 14 00:04:07.436362 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.436283    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4dd826b-f709-4f30-8732-b473baa52021" path="/var/lib/kubelet/pods/f4dd826b-f709-4f30-8732-b473baa52021/volumes"
Jul 14 00:04:07.972234 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.972203    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:07.972614 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.972596    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:07.973235 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:07.973216    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:21.132751 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:21.132707    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:04:21.133272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:21.132992    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy" containerID="cri-o://0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138" gracePeriod=30
Jul 14 00:04:22.971444 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.971406    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:04:22.971947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.971770    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dd826b-f709-4f30-8732-b473baa52021" containerName="maas-api"
Jul 14 00:04:22.971947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.971797    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dd826b-f709-4f30-8732-b473baa52021" containerName="maas-api"
Jul 14 00:04:22.971947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.971873    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dd826b-f709-4f30-8732-b473baa52021" containerName="maas-api"
Jul 14 00:04:22.974582 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.974562    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:22.975480 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.975452    2558 patch_prober.go:28] interesting pod/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 14 00:04:22.975590 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.975518    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 00:04:22.977401 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.977378    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-982b7aec-serving-cert\""
Jul 14 00:04:22.984571 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:22.984548    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:04:23.093161 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.093124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5lh\" (UniqueName: \"kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.093317 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.093189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.093317 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.093268    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.194211 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.194174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.194388 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.194242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.194388 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.194294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5lh\" (UniqueName: \"kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.194388 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:23.194334    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-serving-cert: secret "maas-api-e2e-migrate-982b7aec-serving-cert" not found
Jul 14 00:04:23.194541 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:23.194409    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls podName:2d92d3d0-1ee4-4869-b8dd-b4d991396224 nodeName:}" failed. No retries permitted until 2026-07-14 00:04:23.694386442 +0000 UTC m=+1511.981604768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls") pod "maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" (UID: "2d92d3d0-1ee4-4869-b8dd-b4d991396224") : secret "maas-api-e2e-migrate-982b7aec-serving-cert" not found
Jul 14 00:04:23.194913 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.194873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.203262 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.203241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5lh\" (UniqueName: \"kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.699412 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.699381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.701934 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.701911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") pod \"maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:23.887294 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:23.887250    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:24.021989 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:24.021930    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:04:24.024647 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:04:24.024614    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d92d3d0_1ee4_4869_b8dd_b4d991396224.slice/crio-8b0c232afcbb353a22ca1b0d613d50d277692821dc6756c742beef36635f10b8 WatchSource:0}: Error finding container 8b0c232afcbb353a22ca1b0d613d50d277692821dc6756c742beef36635f10b8: Status 404 returned error can't find the container with id 8b0c232afcbb353a22ca1b0d613d50d277692821dc6756c742beef36635f10b8
Jul 14 00:04:24.139960 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:24.139925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" event={"ID":"2d92d3d0-1ee4-4869-b8dd-b4d991396224","Type":"ContainerStarted","Data":"8b0c232afcbb353a22ca1b0d613d50d277692821dc6756c742beef36635f10b8"}
Jul 14 00:04:25.146217 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:25.146182    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" event={"ID":"2d92d3d0-1ee4-4869-b8dd-b4d991396224","Type":"ContainerStarted","Data":"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"}
Jul 14 00:04:25.146552 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:25.146261    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:25.164268 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:25.164225    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" podStartSLOduration=2.8449200770000003 podStartE2EDuration="3.164211267s" totalImagesPullingTime="319.29119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:04:25.162007287 +0000 UTC m=+1513.449225637" watchObservedRunningTime="2026-07-14 00:04:25.164211267 +0000 UTC m=+1513.451429612"
Jul 14 00:04:26.390469 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.390446    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:26.422338 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422338 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422311    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422367    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422520    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422738 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422546    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6rqn\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422738 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422738 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422579    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:04:26.422738 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data\") pod \"420130a8-38ef-458c-9e54-a56ab38e636c\" (UID: \"420130a8-38ef-458c-9e54-a56ab38e636c\") "
Jul 14 00:04:26.422980 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422765    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:04:26.422980 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422964    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.423081 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.422983    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.423235 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.423096    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:04:26.423235 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.423171    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:04:26.424351 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.424319    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data" (OuterVolumeSpecName: "istio-data") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:04:26.425559 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.425524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token" (OuterVolumeSpecName: "istio-token") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:04:26.425674 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.425642    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:04:26.426436 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.426394    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:04:26.426526 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.426479    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn" (OuterVolumeSpecName: "kube-api-access-h6rqn") pod "420130a8-38ef-458c-9e54-a56ab38e636c" (UID: "420130a8-38ef-458c-9e54-a56ab38e636c"). InnerVolumeSpecName "kube-api-access-h6rqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:04:26.523833 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523806    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.523833 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523834    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.524042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523846    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420130a8-38ef-458c-9e54-a56ab38e636c-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.524042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523857    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.524042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523869    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420130a8-38ef-458c-9e54-a56ab38e636c-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.524042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523881    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rqn\" (UniqueName: \"kubernetes.io/projected/420130a8-38ef-458c-9e54-a56ab38e636c-kube-api-access-h6rqn\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:26.524042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:26.523907    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420130a8-38ef-458c-9e54-a56ab38e636c-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:27.158185 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.158151    2558 generic.go:356] "Generic (PLEG): container finished" podID="420130a8-38ef-458c-9e54-a56ab38e636c" containerID="0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138" exitCode=0
Jul 14 00:04:27.158339 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.158207    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"
Jul 14 00:04:27.158339 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.158206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerDied","Data":"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"}
Jul 14 00:04:27.158339 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.158291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x" event={"ID":"420130a8-38ef-458c-9e54-a56ab38e636c","Type":"ContainerDied","Data":"e3b4fdf4b4e994f949481b3657250fc8fdbecdfb559a6027115e5aef1bae58e1"}
Jul 14 00:04:27.158339 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.158316    2558 scope.go:117] "RemoveContainer" containerID="0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"
Jul 14 00:04:27.167736 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.167719    2558 scope.go:117] "RemoveContainer" containerID="6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"
Jul 14 00:04:27.175175 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.175159    2558 scope.go:117] "RemoveContainer" containerID="0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"
Jul 14 00:04:27.175411 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:27.175391    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138\": container with ID starting with 0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138 not found: ID does not exist" containerID="0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"
Jul 14 00:04:27.175461 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.175419    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138"} err="failed to get container status \"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138\": rpc error: code = NotFound desc = could not find container \"0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138\": container with ID starting with 0172f87c23b1656a1095ae51d5cc483840ca29e41187c941c2a0863852a0d138 not found: ID does not exist"
Jul 14 00:04:27.175461 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.175438    2558 scope.go:117] "RemoveContainer" containerID="6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"
Jul 14 00:04:27.175645 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:27.175630    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d\": container with ID starting with 6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d not found: ID does not exist" containerID="6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"
Jul 14 00:04:27.175686 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.175649    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d"} err="failed to get container status \"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d\": rpc error: code = NotFound desc = could not find container \"6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d\": container with ID starting with 6c97f0f1287850aa97982e00797ce757a3a4457494528c18a056c38632ebaf9d not found: ID does not exist"
Jul 14 00:04:27.180250 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.180229    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:04:27.185607 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.185589    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-30780878-openshift-default-6dc654f6b5-2684x"]
Jul 14 00:04:27.403059 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.403027    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:27.403408 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.403373    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:04:27.403408 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.403384    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:04:27.403479 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.403451    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:04:27.403479 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.403459    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:04:27.406315 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.406298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:27.408741 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.408680    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:04:27.414125 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.414101    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:27.430437 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.430410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6\" (UID: \"d013d32d-32b0-42ac-a41b-8d4088215fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:27.433750 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.433718    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" path="/var/lib/kubelet/pods/420130a8-38ef-458c-9e54-a56ab38e636c/volumes"
Jul 14 00:04:27.531792 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.531755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6\" (UID: \"d013d32d-32b0-42ac-a41b-8d4088215fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:27.532864 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.532828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6\" (UID: \"d013d32d-32b0-42ac-a41b-8d4088215fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:27.718455 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.718383    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:27.847027 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:27.846996    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:27.847689 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:04:27.847660    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd013d32d_32b0_42ac_a41b_8d4088215fcc.slice/crio-7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87 WatchSource:0}: Error finding container 7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87: Status 404 returned error can't find the container with id 7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87
Jul 14 00:04:28.165318 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:28.165287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerStarted","Data":"e03d31d2f32dfa3492c66bc2f88e2ef946624172d0dae3bda83318e304c71367"}
Jul 14 00:04:28.165433 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:28.165337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerStarted","Data":"7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87"}
Jul 14 00:04:28.180689 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:28.180646    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" podStartSLOduration=1.180632026 podStartE2EDuration="1.180632026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:04:28.17887426 +0000 UTC m=+1516.466092607" watchObservedRunningTime="2026-07-14 00:04:28.180632026 +0000 UTC m=+1516.467850373"
Jul 14 00:04:29.172369 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:29.172336    2558 generic.go:356] "Generic (PLEG): container finished" podID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerID="e03d31d2f32dfa3492c66bc2f88e2ef946624172d0dae3bda83318e304c71367" exitCode=7
Jul 14 00:04:29.172734 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:29.172374    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerDied","Data":"e03d31d2f32dfa3492c66bc2f88e2ef946624172d0dae3bda83318e304c71367"}
Jul 14 00:04:29.172734 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:29.172588    2558 scope.go:117] "RemoveContainer" containerID="e03d31d2f32dfa3492c66bc2f88e2ef946624172d0dae3bda83318e304c71367"
Jul 14 00:04:30.179601 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:30.179567    2558 generic.go:356] "Generic (PLEG): container finished" podID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerID="bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6" exitCode=7
Jul 14 00:04:30.180270 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:30.179609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerDied","Data":"bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6"}
Jul 14 00:04:30.180270 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:30.179656    2558 scope.go:117] "RemoveContainer" containerID="e03d31d2f32dfa3492c66bc2f88e2ef946624172d0dae3bda83318e304c71367"
Jul 14 00:04:30.180270 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:30.179844    2558 scope.go:117] "RemoveContainer" containerID="bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6"
Jul 14 00:04:30.180270 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:30.180062    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6_odh-ai-gateway-infra(d013d32d-32b0-42ac-a41b-8d4088215fcc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc"
Jul 14 00:04:31.156836 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:31.156808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:04:31.187016 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:31.186989    2558 scope.go:117] "RemoveContainer" containerID="bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6"
Jul 14 00:04:31.187455 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:04:31.187199    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6_odh-ai-gateway-infra(d013d32d-32b0-42ac-a41b-8d4088215fcc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc"
Jul 14 00:04:46.426817 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:46.426787    2558 scope.go:117] "RemoveContainer" containerID="bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6"
Jul 14 00:04:47.270793 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:47.270757    2558 generic.go:356] "Generic (PLEG): container finished" podID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerID="92ef0a7e2b625e908c3e6f391922c6348713785a74da131dae06a26795a1b645" exitCode=0
Jul 14 00:04:47.270973 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:47.270826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerDied","Data":"92ef0a7e2b625e908c3e6f391922c6348713785a74da131dae06a26795a1b645"}
Jul 14 00:04:47.270973 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:47.270895    2558 scope.go:117] "RemoveContainer" containerID="bc33edea7cbf7cd94dc59772d8ebfd07f8c708f0a8cf37ee62500f3293d9c0a6"
Jul 14 00:04:48.302014 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:48.301983    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:48.413115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:48.413091    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:48.505855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:48.505823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca\") pod \"d013d32d-32b0-42ac-a41b-8d4088215fcc\" (UID: \"d013d32d-32b0-42ac-a41b-8d4088215fcc\") "
Jul 14 00:04:48.506162 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:48.506139    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d013d32d-32b0-42ac-a41b-8d4088215fcc" (UID: "d013d32d-32b0-42ac-a41b-8d4088215fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:04:48.606506 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:48.606445    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013d32d-32b0-42ac-a41b-8d4088215fcc-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:04:49.283031 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.282999    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6" event={"ID":"d013d32d-32b0-42ac-a41b-8d4088215fcc","Type":"ContainerDied","Data":"7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87"}
Jul 14 00:04:49.283031 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.283048    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7572b7af9ff3910f69d908910cca421d74f3147d6b167a7ba8a5b6c7e8060e87"
Jul 14 00:04:49.283234 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.283014    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"
Jul 14 00:04:49.313495 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.313469    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:49.315473 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.315453    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-x5fc6"]
Jul 14 00:04:49.431777 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:04:49.431738    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" path="/var/lib/kubelet/pods/d013d32d-32b0-42ac-a41b-8d4088215fcc/volumes"
Jul 14 00:05:27.448358 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448328    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"]
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448664    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448675    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="420130a8-38ef-458c-9e54-a56ab38e636c" containerName="istio-proxy"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448684    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448689    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448704    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448709    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448715    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.448772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448720    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.449059 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448781    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.449059 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.448790    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:27.451608 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.451591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:27.453957 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.453934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:05:27.458605 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.458582    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"]
Jul 14 00:05:27.520203 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.520173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg\" (UID: \"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:27.621078 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.621050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg\" (UID: \"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:27.621622 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.621605    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg\" (UID: \"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:27.765398 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.765367    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:27.892670 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:27.892646    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"]
Jul 14 00:05:27.896098 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:05:27.896061    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f0d55f_4e6f_4c1a_8e7f_dc854aea78f6.slice/crio-e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3 WatchSource:0}: Error finding container e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3: Status 404 returned error can't find the container with id e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3
Jul 14 00:05:28.469763 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:28.469672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg" event={"ID":"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6","Type":"ContainerStarted","Data":"ce84d7d1982d4deaeba877a5b1244bec3345a472add10397ad287afe9b45740e"}
Jul 14 00:05:28.469763 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:28.469726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg" event={"ID":"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6","Type":"ContainerStarted","Data":"e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3"}
Jul 14 00:05:28.485125 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:28.485078    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg" podStartSLOduration=1.485062367 podStartE2EDuration="1.485062367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:05:28.484441353 +0000 UTC m=+1576.771659700" watchObservedRunningTime="2026-07-14 00:05:28.485062367 +0000 UTC m=+1576.772280712"
Jul 14 00:05:29.476402 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:29.476367    2558 generic.go:356] "Generic (PLEG): container finished" podID="47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6" containerID="ce84d7d1982d4deaeba877a5b1244bec3345a472add10397ad287afe9b45740e" exitCode=0
Jul 14 00:05:29.476791 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:29.476420    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg" event={"ID":"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6","Type":"ContainerDied","Data":"ce84d7d1982d4deaeba877a5b1244bec3345a472add10397ad287afe9b45740e"}
Jul 14 00:05:30.614394 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:30.614369    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:30.639712 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:30.639688    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca\") pod \"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6\" (UID: \"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6\") "
Jul 14 00:05:30.640073 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:30.640051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6" (UID: "47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:05:30.741229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:30.741148    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:05:31.487415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:31.487384    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg"
Jul 14 00:05:31.487573 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:31.487392    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg" event={"ID":"47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6","Type":"ContainerDied","Data":"e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3"}
Jul 14 00:05:31.487573 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:31.487500    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f787d9c3718a6502435ca886311e0f43606ac79a2a0f570d981fdf81b87cb3"
Jul 14 00:05:32.896024 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:32.895991    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:05:32.896421 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:32.896244    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" podUID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" containerName="maas-api" containerID="cri-o://ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd" gracePeriod=30
Jul 14 00:05:33.147440 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.147418    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:05:33.159760 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.159743    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") pod \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") "
Jul 14 00:05:33.159835 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.159770    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw5lh\" (UniqueName: \"kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh\") pod \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") "
Jul 14 00:05:33.159835 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.159819    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca\") pod \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\" (UID: \"2d92d3d0-1ee4-4869-b8dd-b4d991396224\") "
Jul 14 00:05:33.160190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.160158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d92d3d0-1ee4-4869-b8dd-b4d991396224" (UID: "2d92d3d0-1ee4-4869-b8dd-b4d991396224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:05:33.161688 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.161664    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d92d3d0-1ee4-4869-b8dd-b4d991396224" (UID: "2d92d3d0-1ee4-4869-b8dd-b4d991396224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:05:33.161796 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.161777    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh" (OuterVolumeSpecName: "kube-api-access-tw5lh") pod "2d92d3d0-1ee4-4869-b8dd-b4d991396224" (UID: "2d92d3d0-1ee4-4869-b8dd-b4d991396224"). InnerVolumeSpecName "kube-api-access-tw5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:05:33.261285 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.261261    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d92d3d0-1ee4-4869-b8dd-b4d991396224-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:05:33.261285 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.261284    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d92d3d0-1ee4-4869-b8dd-b4d991396224-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:05:33.261456 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.261294    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5lh\" (UniqueName: \"kubernetes.io/projected/2d92d3d0-1ee4-4869-b8dd-b4d991396224-kube-api-access-tw5lh\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:05:33.496857 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.496821    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" containerID="ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd" exitCode=0
Jul 14 00:05:33.497025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.496867    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"
Jul 14 00:05:33.497025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.496927    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" event={"ID":"2d92d3d0-1ee4-4869-b8dd-b4d991396224","Type":"ContainerDied","Data":"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"}
Jul 14 00:05:33.497025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.496980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf" event={"ID":"2d92d3d0-1ee4-4869-b8dd-b4d991396224","Type":"ContainerDied","Data":"8b0c232afcbb353a22ca1b0d613d50d277692821dc6756c742beef36635f10b8"}
Jul 14 00:05:33.497025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.497002    2558 scope.go:117] "RemoveContainer" containerID="ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"
Jul 14 00:05:33.504973 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.504953    2558 scope.go:117] "RemoveContainer" containerID="ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"
Jul 14 00:05:33.505246 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:05:33.505225    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd\": container with ID starting with ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd not found: ID does not exist" containerID="ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"
Jul 14 00:05:33.505328 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.505256    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd"} err="failed to get container status \"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd\": rpc error: code = NotFound desc = could not find container \"ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd\": container with ID starting with ac71ba396fc1dde21c0180635c780e534a2026b9c2c2f8fb80a96bf8a3aa5bdd not found: ID does not exist"
Jul 14 00:05:33.513198 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.513173    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:05:33.516730 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:33.516707    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-982b7aec-bff8888f8-vwsgf"]
Jul 14 00:05:35.431612 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:35.431578    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" path="/var/lib/kubelet/pods/2d92d3d0-1ee4-4869-b8dd-b4d991396224/volumes"
Jul 14 00:05:39.450425 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.450390    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:05:39.450978 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.450957    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" containerName="maas-api"
Jul 14 00:05:39.451025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.450982    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" containerName="maas-api"
Jul 14 00:05:39.451025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.451005    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6" containerName="revoke-keys"
Jul 14 00:05:39.451025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.451014    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6" containerName="revoke-keys"
Jul 14 00:05:39.451131 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.451116    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013d32d-32b0-42ac-a41b-8d4088215fcc" containerName="revoke-keys"
Jul 14 00:05:39.451167 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.451130    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6" containerName="revoke-keys"
Jul 14 00:05:39.451167 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.451141    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d92d3d0-1ee4-4869-b8dd-b4d991396224" containerName="maas-api"
Jul 14 00:05:39.454541 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.454522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.457641 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.457611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0bcce182-openshift-default-dockercfg-4sd85\""
Jul 14 00:05:39.465499 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.465478    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:05:39.513593 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513747 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513747 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513747 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513975 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513975 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.513975 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.514133 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.513978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.514133 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.514046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btxhc\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.614981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.614940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615174 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.614999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615174 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615295 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615295 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615295 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxhc\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615456 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615456 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615554 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615615 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615670 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615670 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.615933 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.616024 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.615933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.617450 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.617417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.617568 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.617521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.623601 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.623574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.624447 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.624421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxhc\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc\") pod \"e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") " pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.766338 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.766318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:39.901556 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.901532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:05:39.903301 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:05:39.903268    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440ce142_922a_43f5_9310_ccee53b4c0c3.slice/crio-aa28468b13b89d7da81aee04be0168b862797d1eeb9117164182c69d16be4b4a WatchSource:0}: Error finding container aa28468b13b89d7da81aee04be0168b862797d1eeb9117164182c69d16be4b4a: Status 404 returned error can't find the container with id aa28468b13b89d7da81aee04be0168b862797d1eeb9117164182c69d16be4b4a
Jul 14 00:05:39.905542 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.905509    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:39.905638 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.905581    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:39.905638 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:39.905606    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:40.531984 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.531941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerStarted","Data":"1ab388930d2bb4d6b36cabcd4f96c4f1fb2fcb1ba9b51f83c9a42dd530b1cb8e"}
Jul 14 00:05:40.532383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.532005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerStarted","Data":"aa28468b13b89d7da81aee04be0168b862797d1eeb9117164182c69d16be4b4a"}
Jul 14 00:05:40.560112 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.560058    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podStartSLOduration=1.5600446780000001 podStartE2EDuration="1.560044678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:05:40.557499301 +0000 UTC m=+1588.844717640" watchObservedRunningTime="2026-07-14 00:05:40.560044678 +0000 UTC m=+1588.847263023"
Jul 14 00:05:40.767040 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.766997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:40.771634 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.771608    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:40.785606 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.785531    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:05:40.787928 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.787909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.790584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.790566    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0bcce182-serving-cert\""
Jul 14 00:05:40.801843 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.801818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:05:40.825433 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.825406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.825533 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.825474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.825533 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.825511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d88z\" (UniqueName: \"kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.926771 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.926730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.927010 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.926774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d88z\" (UniqueName: \"kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.927010 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.926865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.927010 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:05:40.926999    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-serving-cert: secret "maas-api-e2e-ait-0bcce182-serving-cert" not found
Jul 14 00:05:40.927191 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:05:40.927074    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls podName:572023f5-649f-455a-b4af-8d3c61a324c6 nodeName:}" failed. No retries permitted until 2026-07-14 00:05:41.427049336 +0000 UTC m=+1589.714267660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls") pod "maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" (UID: "572023f5-649f-455a-b4af-8d3c61a324c6") : secret "maas-api-e2e-ait-0bcce182-serving-cert" not found
Jul 14 00:05:40.927404 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.927383    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:40.938667 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:40.938638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d88z\" (UniqueName: \"kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:41.432600 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.432572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:41.434929 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.434905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") pod \"maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:41.537796 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.537761    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:41.538686 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.538668    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:41.700910 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.700811    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:41.829689 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.829655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:05:41.831218 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:05:41.831172    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572023f5_649f_455a_b4af_8d3c61a324c6.slice/crio-e86f2a522fbf345b71a8e80007b49b47fd7284457c9fd125a8aac12bc2d51bfe WatchSource:0}: Error finding container e86f2a522fbf345b71a8e80007b49b47fd7284457c9fd125a8aac12bc2d51bfe: Status 404 returned error can't find the container with id e86f2a522fbf345b71a8e80007b49b47fd7284457c9fd125a8aac12bc2d51bfe
Jul 14 00:05:41.833025 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:41.833002    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:05:42.545177 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:42.545137    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" event={"ID":"572023f5-649f-455a-b4af-8d3c61a324c6","Type":"ContainerStarted","Data":"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"}
Jul 14 00:05:42.545558 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:42.545200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" event={"ID":"572023f5-649f-455a-b4af-8d3c61a324c6","Type":"ContainerStarted","Data":"e86f2a522fbf345b71a8e80007b49b47fd7284457c9fd125a8aac12bc2d51bfe"}
Jul 14 00:05:42.545558 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:42.545401    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:42.563180 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:42.563134    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" podStartSLOduration=2.066342841 podStartE2EDuration="2.563119459s" totalImagesPullingTime="496.776618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:05:42.560936071 +0000 UTC m=+1590.848154417" watchObservedRunningTime="2026-07-14 00:05:42.563119459 +0000 UTC m=+1590.850337804"
Jul 14 00:05:48.555646 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:48.555615    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:05:51.590144 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.590058    2558 generic.go:356] "Generic (PLEG): container finished" podID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerID="1ab388930d2bb4d6b36cabcd4f96c4f1fb2fcb1ba9b51f83c9a42dd530b1cb8e" exitCode=0
Jul 14 00:05:51.590556 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.590135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerDied","Data":"1ab388930d2bb4d6b36cabcd4f96c4f1fb2fcb1ba9b51f83c9a42dd530b1cb8e"}
Jul 14 00:05:51.590556 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.590526    2558 scope.go:117] "RemoveContainer" containerID="1ab388930d2bb4d6b36cabcd4f96c4f1fb2fcb1ba9b51f83c9a42dd530b1cb8e"
Jul 14 00:05:51.591624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.591596    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:51.591702 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.591665    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:51.591702 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:51.591698    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:05:52.596946 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:52.596912    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerStarted","Data":"75c7cb5a12c91a8ec647a40ada335abffd3e05f473861aa3c3728f3bea283366"}
Jul 14 00:05:52.767153 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:52.767113    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:52.768188 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:52.768165    2558 patch_prober.go:28] interesting pod/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 14 00:05:52.768304 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:52.768215    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 14 00:05:53.767378 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:53.767344    2558 patch_prober.go:28] interesting pod/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 14 00:05:53.767844 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:53.767412    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 14 00:05:54.766994 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:54.766954    2558 patch_prober.go:28] interesting pod/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 14 00:05:54.767198 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:54.767020    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 14 00:05:55.766798 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:55.766762    2558 patch_prober.go:28] interesting pod/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 14 00:05:55.767213 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:55.766816    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 14 00:05:56.538341 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:56.538307    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:56.770294 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:56.770261    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:05:56.771071 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:05:56.771050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:06:56.400562 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.400507    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:06:56.401412 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.401377    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" podUID="572023f5-649f-455a-b4af-8d3c61a324c6" containerName="maas-api" containerID="cri-o://06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3" gracePeriod=30
Jul 14 00:06:56.672677 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.672652    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:06:56.775981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.775957    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d88z\" (UniqueName: \"kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z\") pod \"572023f5-649f-455a-b4af-8d3c61a324c6\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") "
Jul 14 00:06:56.776161 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.776000    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca\") pod \"572023f5-649f-455a-b4af-8d3c61a324c6\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") "
Jul 14 00:06:56.776161 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.776056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") pod \"572023f5-649f-455a-b4af-8d3c61a324c6\" (UID: \"572023f5-649f-455a-b4af-8d3c61a324c6\") "
Jul 14 00:06:56.776454 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.776417    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "572023f5-649f-455a-b4af-8d3c61a324c6" (UID: "572023f5-649f-455a-b4af-8d3c61a324c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:06:56.778087 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.778068    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z" (OuterVolumeSpecName: "kube-api-access-2d88z") pod "572023f5-649f-455a-b4af-8d3c61a324c6" (UID: "572023f5-649f-455a-b4af-8d3c61a324c6"). InnerVolumeSpecName "kube-api-access-2d88z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:06:56.778156 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.778134    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "572023f5-649f-455a-b4af-8d3c61a324c6" (UID: "572023f5-649f-455a-b4af-8d3c61a324c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:06:56.877422 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.877389    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d88z\" (UniqueName: \"kubernetes.io/projected/572023f5-649f-455a-b4af-8d3c61a324c6-kube-api-access-2d88z\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:06:56.877422 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.877414    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/572023f5-649f-455a-b4af-8d3c61a324c6-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:06:56.877422 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.877428    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/572023f5-649f-455a-b4af-8d3c61a324c6-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:06:56.907624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.907550    2558 generic.go:356] "Generic (PLEG): container finished" podID="572023f5-649f-455a-b4af-8d3c61a324c6" containerID="06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3" exitCode=0
Jul 14 00:06:56.907624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.907588    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" event={"ID":"572023f5-649f-455a-b4af-8d3c61a324c6","Type":"ContainerDied","Data":"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"}
Jul 14 00:06:56.907624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.907594    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"
Jul 14 00:06:56.907799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.907635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5" event={"ID":"572023f5-649f-455a-b4af-8d3c61a324c6","Type":"ContainerDied","Data":"e86f2a522fbf345b71a8e80007b49b47fd7284457c9fd125a8aac12bc2d51bfe"}
Jul 14 00:06:56.907799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.907660    2558 scope.go:117] "RemoveContainer" containerID="06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"
Jul 14 00:06:56.916485 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.916468    2558 scope.go:117] "RemoveContainer" containerID="06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"
Jul 14 00:06:56.916723 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:06:56.916705    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3\": container with ID starting with 06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3 not found: ID does not exist" containerID="06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"
Jul 14 00:06:56.916777 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.916731    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3"} err="failed to get container status \"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3\": rpc error: code = NotFound desc = could not find container \"06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3\": container with ID starting with 06ab4d55a43359929072626f3fe2458ece67225649d19a1d6542b773c6aeeca3 not found: ID does not exist"
Jul 14 00:06:56.927066 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.927038    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:06:56.930172 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:56.930150    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0bcce182-7c96d758cb-gpxn5"]
Jul 14 00:06:57.433118 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:06:57.433085    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572023f5-649f-455a-b4af-8d3c61a324c6" path="/var/lib/kubelet/pods/572023f5-649f-455a-b4af-8d3c61a324c6/volumes"
Jul 14 00:07:11.844589 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:11.844551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:07:11.845192 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:11.844746    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy" containerID="cri-o://75c7cb5a12c91a8ec647a40ada335abffd3e05f473861aa3c3728f3bea283366" gracePeriod=30
Jul 14 00:07:13.368258 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.368217    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:07:13.368784 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.368764    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572023f5-649f-455a-b4af-8d3c61a324c6" containerName="maas-api"
Jul 14 00:07:13.368866 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.368787    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="572023f5-649f-455a-b4af-8d3c61a324c6" containerName="maas-api"
Jul 14 00:07:13.368943 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.368907    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="572023f5-649f-455a-b4af-8d3c61a324c6" containerName="maas-api"
Jul 14 00:07:13.371789 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.371770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.374346 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.374323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bb015d31-serving-cert\""
Jul 14 00:07:13.381520 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.381493    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:07:13.423528 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.423487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.423726 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.423668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmbn\" (UniqueName: \"kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.423861 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.423750    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.524272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.524227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.524494 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.524313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmbn\" (UniqueName: \"kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.524494 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.524338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.524494 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:07:13.524446    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-serving-cert: secret "maas-api-e2e-derive-bb015d31-serving-cert" not found
Jul 14 00:07:13.524667 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:07:13.524510    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls podName:f8b94662-5807-468f-81b3-8a65c71cb51d nodeName:}" failed. No retries permitted until 2026-07-14 00:07:14.024492327 +0000 UTC m=+1682.311710651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls") pod "maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" (UID: "f8b94662-5807-468f-81b3-8a65c71cb51d") : secret "maas-api-e2e-derive-bb015d31-serving-cert" not found
Jul 14 00:07:13.525004 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.524980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:13.534933 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:13.534905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmbn\" (UniqueName: \"kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:14.029717 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:14.029681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:14.032191 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:14.032164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") pod \"maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:14.283542 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:14.283460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:14.420284 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:14.420251    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:07:14.422244 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:07:14.422203    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b94662_5807_468f_81b3_8a65c71cb51d.slice/crio-fef237b139f80ce1c89e93cbb242a6613eecd67244306472c5177f2192947fad WatchSource:0}: Error finding container fef237b139f80ce1c89e93cbb242a6613eecd67244306472c5177f2192947fad: Status 404 returned error can't find the container with id fef237b139f80ce1c89e93cbb242a6613eecd67244306472c5177f2192947fad
Jul 14 00:07:15.001497 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:15.001465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" event={"ID":"f8b94662-5807-468f-81b3-8a65c71cb51d","Type":"ContainerStarted","Data":"28633b9c312335d41a80b0d1c98bc7fb5eb1b9001ac0b551226405b8108038ce"}
Jul 14 00:07:15.001648 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:15.001517    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" event={"ID":"f8b94662-5807-468f-81b3-8a65c71cb51d","Type":"ContainerStarted","Data":"fef237b139f80ce1c89e93cbb242a6613eecd67244306472c5177f2192947fad"}
Jul 14 00:07:15.001648 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:15.001632    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:15.021182 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:15.021118    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" podStartSLOduration=1.516954763 podStartE2EDuration="2.021098079s" totalImagesPullingTime="504.143316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:07:15.018978195 +0000 UTC m=+1683.306196543" watchObservedRunningTime="2026-07-14 00:07:15.021098079 +0000 UTC m=+1683.308316427"
Jul 14 00:07:17.014243 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.014189    2558 generic.go:356] "Generic (PLEG): container finished" podID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerID="75c7cb5a12c91a8ec647a40ada335abffd3e05f473861aa3c3728f3bea283366" exitCode=0
Jul 14 00:07:17.014637 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.014341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerDied","Data":"75c7cb5a12c91a8ec647a40ada335abffd3e05f473861aa3c3728f3bea283366"}
Jul 14 00:07:17.014637 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.014442    2558 scope.go:117] "RemoveContainer" containerID="1ab388930d2bb4d6b36cabcd4f96c4f1fb2fcb1ba9b51f83c9a42dd530b1cb8e"
Jul 14 00:07:17.105517 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.105457    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:07:17.158471 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158444    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158471 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158474    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btxhc\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158521    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158566    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158609    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.158693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.159070 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.158762    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:07:17.159070 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159040    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:07:17.159186 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token\") pod \"440ce142-922a-43f5-9310-ccee53b4c0c3\" (UID: \"440ce142-922a-43f5-9310-ccee53b4c0c3\") "
Jul 14 00:07:17.159186 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:07:17.159295 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:07:17.159295 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159270    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:07:17.159429 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159403    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.159491 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159431    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.159491 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159446    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ce142-922a-43f5-9310-ccee53b4c0c3-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.159491 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159459    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.159491 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.159473    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.160788 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.160758    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc" (OuterVolumeSpecName: "kube-api-access-btxhc") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "kube-api-access-btxhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:07:17.161090 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.161063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:07:17.161196 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.161093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:07:17.161372 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.161344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "440ce142-922a-43f5-9310-ccee53b4c0c3" (UID: "440ce142-922a-43f5-9310-ccee53b4c0c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:07:17.260780 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.260736    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.260780 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.260771    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.260780 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.260784    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btxhc\" (UniqueName: \"kubernetes.io/projected/440ce142-922a-43f5-9310-ccee53b4c0c3-kube-api-access-btxhc\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.261047 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.260797    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ce142-922a-43f5-9310-ccee53b4c0c3-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:17.677319 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677277    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:17.677809 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677794    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:07:17.677866 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677814    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:07:17.677866 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677844    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:07:17.677866 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677853    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:07:17.677985 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.677953    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:07:17.682497 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.682479    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:17.685221 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.685195    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:07:17.687000 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.686975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:17.764829 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.764795    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr\" (UID: \"a87e39e7-4924-412d-8bc2-863a665816b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:17.866136 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.866101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr\" (UID: \"a87e39e7-4924-412d-8bc2-863a665816b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:17.866780 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.866761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr\" (UID: \"a87e39e7-4924-412d-8bc2-863a665816b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:17.994510 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:17.994461    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:18.020619 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.020550    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48" event={"ID":"440ce142-922a-43f5-9310-ccee53b4c0c3","Type":"ContainerDied","Data":"aa28468b13b89d7da81aee04be0168b862797d1eeb9117164182c69d16be4b4a"}
Jul 14 00:07:18.020619 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.020579    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"
Jul 14 00:07:18.020619 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.020607    2558 scope.go:117] "RemoveContainer" containerID="75c7cb5a12c91a8ec647a40ada335abffd3e05f473861aa3c3728f3bea283366"
Jul 14 00:07:18.040551 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.040494    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:07:18.042532 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.042509    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0bcce182-openshift-default-5c8b7b65ff-s7h48"]
Jul 14 00:07:18.132661 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:18.132637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:18.134544 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:07:18.134504    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87e39e7_4924_412d_8bc2_863a665816b6.slice/crio-536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79 WatchSource:0}: Error finding container 536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79: Status 404 returned error can't find the container with id 536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79
Jul 14 00:07:19.028212 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:19.028175    2558 generic.go:356] "Generic (PLEG): container finished" podID="a87e39e7-4924-412d-8bc2-863a665816b6" containerID="18cfa5b2ac435bb4d747ae3248f03d3048087ee73c9d46c9849f6b134d84b11e" exitCode=7
Jul 14 00:07:19.028591 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:19.028264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" event={"ID":"a87e39e7-4924-412d-8bc2-863a665816b6","Type":"ContainerDied","Data":"18cfa5b2ac435bb4d747ae3248f03d3048087ee73c9d46c9849f6b134d84b11e"}
Jul 14 00:07:19.028591 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:19.028321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" event={"ID":"a87e39e7-4924-412d-8bc2-863a665816b6","Type":"ContainerStarted","Data":"536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79"}
Jul 14 00:07:19.028591 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:19.028415    2558 scope.go:117] "RemoveContainer" containerID="18cfa5b2ac435bb4d747ae3248f03d3048087ee73c9d46c9849f6b134d84b11e"
Jul 14 00:07:19.432271 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:19.432239    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" path="/var/lib/kubelet/pods/440ce142-922a-43f5-9310-ccee53b4c0c3/volumes"
Jul 14 00:07:20.035758 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:20.035721    2558 generic.go:356] "Generic (PLEG): container finished" podID="a87e39e7-4924-412d-8bc2-863a665816b6" containerID="78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc" exitCode=7
Jul 14 00:07:20.036278 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:20.035821    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" event={"ID":"a87e39e7-4924-412d-8bc2-863a665816b6","Type":"ContainerDied","Data":"78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc"}
Jul 14 00:07:20.036278 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:20.035916    2558 scope.go:117] "RemoveContainer" containerID="18cfa5b2ac435bb4d747ae3248f03d3048087ee73c9d46c9849f6b134d84b11e"
Jul 14 00:07:20.036278 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:20.036072    2558 scope.go:117] "RemoveContainer" containerID="78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc"
Jul 14 00:07:20.036467 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:07:20.036323    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr_odh-ai-gateway-infra(a87e39e7-4924-412d-8bc2-863a665816b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" podUID="a87e39e7-4924-412d-8bc2-863a665816b6"
Jul 14 00:07:21.012532 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:21.012505    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:07:21.046147 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:21.046117    2558 scope.go:117] "RemoveContainer" containerID="78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc"
Jul 14 00:07:21.046508 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:07:21.046366    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr_odh-ai-gateway-infra(a87e39e7-4924-412d-8bc2-863a665816b6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" podUID="a87e39e7-4924-412d-8bc2-863a665816b6"
Jul 14 00:07:31.426951 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:31.426910    2558 scope.go:117] "RemoveContainer" containerID="78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc"
Jul 14 00:07:32.099357 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:32.099327    2558 generic.go:356] "Generic (PLEG): container finished" podID="a87e39e7-4924-412d-8bc2-863a665816b6" containerID="41c74872ce413c09925710e0c7a202527010b235d1ea30f028ccce6d30e2887c" exitCode=0
Jul 14 00:07:32.099517 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:32.099373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" event={"ID":"a87e39e7-4924-412d-8bc2-863a665816b6","Type":"ContainerDied","Data":"41c74872ce413c09925710e0c7a202527010b235d1ea30f028ccce6d30e2887c"}
Jul 14 00:07:32.099517 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:32.099419    2558 scope.go:117] "RemoveContainer" containerID="78f5b23bad0621b36b78cfb2c715a19b137a53f5de39ad407e21c78371319fbc"
Jul 14 00:07:33.139433 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:33.139395    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:33.243315 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:33.243294    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:33.295678 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:33.295644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca\") pod \"a87e39e7-4924-412d-8bc2-863a665816b6\" (UID: \"a87e39e7-4924-412d-8bc2-863a665816b6\") "
Jul 14 00:07:33.296051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:33.296026    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a87e39e7-4924-412d-8bc2-863a665816b6" (UID: "a87e39e7-4924-412d-8bc2-863a665816b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:07:33.397094 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:33.397004    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87e39e7-4924-412d-8bc2-863a665816b6-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:07:34.111101 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:34.111026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr" event={"ID":"a87e39e7-4924-412d-8bc2-863a665816b6","Type":"ContainerDied","Data":"536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79"}
Jul 14 00:07:34.111101 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:34.111057    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"
Jul 14 00:07:34.111101 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:34.111103    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536e62a1a091a98c40f4413756445268716efb5ea6a087cb25a06b7f65310c79"
Jul 14 00:07:34.139729 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:34.139696    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:34.143318 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:34.143294    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-wd6rr"]
Jul 14 00:07:35.432536 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:07:35.432495    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" path="/var/lib/kubelet/pods/a87e39e7-4924-412d-8bc2-863a665816b6/volumes"
Jul 14 00:08:07.717393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717354    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"]
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717734    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717745    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717762    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717768    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717828    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717840    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="440ce142-922a-43f5-9310-ccee53b4c0c3" containerName="istio-proxy"
Jul 14 00:08:07.717936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.717846    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:08:07.721076 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.721058    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:07.723468 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.723445    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:08:07.729303 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.729273    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"]
Jul 14 00:08:07.786950 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.786921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-6992j\" (UID: \"7ddc53f5-5906-408a-8b7b-88cb970adc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:07.888072 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.888038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-6992j\" (UID: \"7ddc53f5-5906-408a-8b7b-88cb970adc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:07.888688 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:07.888669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bb015d31-6992j\" (UID: \"7ddc53f5-5906-408a-8b7b-88cb970adc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:08.033221 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:08.033193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:08.168715 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:08.168686    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"]
Jul 14 00:08:08.170455 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:08:08.170419    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddc53f5_5906_408a_8b7b_88cb970adc70.slice/crio-5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b WatchSource:0}: Error finding container 5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b: Status 404 returned error can't find the container with id 5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b
Jul 14 00:08:08.274563 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:08.274529    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j" event={"ID":"7ddc53f5-5906-408a-8b7b-88cb970adc70","Type":"ContainerStarted","Data":"5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b"}
Jul 14 00:08:09.280613 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:09.280574    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ddc53f5-5906-408a-8b7b-88cb970adc70" containerID="21869f2bc14aa95c04edfd8325c5bf46bbe2f9ed9d7614b65bc41525611998c9" exitCode=0
Jul 14 00:08:09.281061 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:09.280630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j" event={"ID":"7ddc53f5-5906-408a-8b7b-88cb970adc70","Type":"ContainerDied","Data":"21869f2bc14aa95c04edfd8325c5bf46bbe2f9ed9d7614b65bc41525611998c9"}
Jul 14 00:08:10.416223 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:10.416201    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:10.509691 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:10.509665    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca\") pod \"7ddc53f5-5906-408a-8b7b-88cb970adc70\" (UID: \"7ddc53f5-5906-408a-8b7b-88cb970adc70\") "
Jul 14 00:08:10.510056 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:10.510035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ddc53f5-5906-408a-8b7b-88cb970adc70" (UID: "7ddc53f5-5906-408a-8b7b-88cb970adc70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:08:10.611166 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:10.611097    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddc53f5-5906-408a-8b7b-88cb970adc70-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:08:11.292005 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:11.291976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j"
Jul 14 00:08:11.292190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:11.292027    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bb015d31-6992j" event={"ID":"7ddc53f5-5906-408a-8b7b-88cb970adc70","Type":"ContainerDied","Data":"5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b"}
Jul 14 00:08:11.292190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:11.292073    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b70e8cdad0e5200aacce57016df2178bf5e30f53a4ed5939249885fd9c5cb5b"
Jul 14 00:08:18.109009 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.108978    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:08:18.109411 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.109181    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" podUID="f8b94662-5807-468f-81b3-8a65c71cb51d" containerName="maas-api" containerID="cri-o://28633b9c312335d41a80b0d1c98bc7fb5eb1b9001ac0b551226405b8108038ce" gracePeriod=30
Jul 14 00:08:18.325534 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.325502    2558 generic.go:356] "Generic (PLEG): container finished" podID="f8b94662-5807-468f-81b3-8a65c71cb51d" containerID="28633b9c312335d41a80b0d1c98bc7fb5eb1b9001ac0b551226405b8108038ce" exitCode=0
Jul 14 00:08:18.325713 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.325569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" event={"ID":"f8b94662-5807-468f-81b3-8a65c71cb51d","Type":"ContainerDied","Data":"28633b9c312335d41a80b0d1c98bc7fb5eb1b9001ac0b551226405b8108038ce"}
Jul 14 00:08:18.361642 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.361585    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:08:18.480648 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.480619    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmbn\" (UniqueName: \"kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn\") pod \"f8b94662-5807-468f-81b3-8a65c71cb51d\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") "
Jul 14 00:08:18.480805 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.480706    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca\") pod \"f8b94662-5807-468f-81b3-8a65c71cb51d\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") "
Jul 14 00:08:18.480805 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.480741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") pod \"f8b94662-5807-468f-81b3-8a65c71cb51d\" (UID: \"f8b94662-5807-468f-81b3-8a65c71cb51d\") "
Jul 14 00:08:18.481091 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.481068    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8b94662-5807-468f-81b3-8a65c71cb51d" (UID: "f8b94662-5807-468f-81b3-8a65c71cb51d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:08:18.482563 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.482531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn" (OuterVolumeSpecName: "kube-api-access-5vmbn") pod "f8b94662-5807-468f-81b3-8a65c71cb51d" (UID: "f8b94662-5807-468f-81b3-8a65c71cb51d"). InnerVolumeSpecName "kube-api-access-5vmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:08:18.482662 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.482598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8b94662-5807-468f-81b3-8a65c71cb51d" (UID: "f8b94662-5807-468f-81b3-8a65c71cb51d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:08:18.582319 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.582282    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmbn\" (UniqueName: \"kubernetes.io/projected/f8b94662-5807-468f-81b3-8a65c71cb51d-kube-api-access-5vmbn\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:08:18.582319 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.582310    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b94662-5807-468f-81b3-8a65c71cb51d-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:08:18.582319 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:18.582320    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8b94662-5807-468f-81b3-8a65c71cb51d-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:08:19.331498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.331467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl" event={"ID":"f8b94662-5807-468f-81b3-8a65c71cb51d","Type":"ContainerDied","Data":"fef237b139f80ce1c89e93cbb242a6613eecd67244306472c5177f2192947fad"}
Jul 14 00:08:19.331498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.331485    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"
Jul 14 00:08:19.332026 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.331529    2558 scope.go:117] "RemoveContainer" containerID="28633b9c312335d41a80b0d1c98bc7fb5eb1b9001ac0b551226405b8108038ce"
Jul 14 00:08:19.353382 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.353357    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:08:19.356069 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.356047    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bb015d31-5c4b547c67-hnwzl"]
Jul 14 00:08:19.432037 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:08:19.432016    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b94662-5807-468f-81b3-8a65c71cb51d" path="/var/lib/kubelet/pods/f8b94662-5807-468f-81b3-8a65c71cb51d/volumes"
Jul 14 00:10:07.765672 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.765639    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:07.766182 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766165    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b94662-5807-468f-81b3-8a65c71cb51d" containerName="maas-api"
Jul 14 00:10:07.766230 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766184    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b94662-5807-468f-81b3-8a65c71cb51d" containerName="maas-api"
Jul 14 00:10:07.766230 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766205    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:10:07.766230 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766214    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:10:07.766329 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766237    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddc53f5-5906-408a-8b7b-88cb970adc70" containerName="revoke-keys"
Jul 14 00:10:07.766329 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766246    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddc53f5-5906-408a-8b7b-88cb970adc70" containerName="revoke-keys"
Jul 14 00:10:07.766392 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766328    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddc53f5-5906-408a-8b7b-88cb970adc70" containerName="revoke-keys"
Jul 14 00:10:07.766392 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766340    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87e39e7-4924-412d-8bc2-863a665816b6" containerName="revoke-keys"
Jul 14 00:10:07.766392 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.766350    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b94662-5807-468f-81b3-8a65c71cb51d" containerName="maas-api"
Jul 14 00:10:07.769577 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.769556    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.772175 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.772153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-45127b63-openshift-default-dockercfg-l5gr5\""
Jul 14 00:10:07.781112 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.780987    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:07.898724 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.898912 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv6w9\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.898912 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.898912 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.898912 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.899097 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.899097 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.898970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.899097 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.899006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.899097 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.899062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999696 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999859 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999859 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999755    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv6w9\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999859 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999859 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:07.999859 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000150 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000150 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000150 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:07.999982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000312 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.000151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000364 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.000310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000407 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.000391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000450 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.000422    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.000949 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.000922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.002265 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.002244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.002548 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.002529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.008330 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.008308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.008442 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.008397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv6w9\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9\") pod \"e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") " pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.083644 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.083560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:08.220643 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.220616    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:08.223304 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:10:08.223260    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf37ef9c9_33a5_4ce7_ad5e_9c21c31a4d04.slice/crio-8379ba0c44c11603d50177949f44a3c9eb41f7bcfebe793e116f6e15b16c32cb WatchSource:0}: Error finding container 8379ba0c44c11603d50177949f44a3c9eb41f7bcfebe793e116f6e15b16c32cb: Status 404 returned error can't find the container with id 8379ba0c44c11603d50177949f44a3c9eb41f7bcfebe793e116f6e15b16c32cb
Jul 14 00:10:08.225472 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.225437    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:08.225584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.225517    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:08.225584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.225562    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:08.855881 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.855848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerStarted","Data":"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"}
Jul 14 00:10:08.856286 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.855929    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerStarted","Data":"8379ba0c44c11603d50177949f44a3c9eb41f7bcfebe793e116f6e15b16c32cb"}
Jul 14 00:10:08.877968 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:08.877920    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podStartSLOduration=1.877906509 podStartE2EDuration="1.877906509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:10:08.876211855 +0000 UTC m=+1857.163430202" watchObservedRunningTime="2026-07-14 00:10:08.877906509 +0000 UTC m=+1857.165124854"
Jul 14 00:10:09.083772 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:09.083737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:09.088340 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:09.088318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:09.861595 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:09.861565    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:09.862496 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:09.862476    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:11.077182 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.077142    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:11.081245 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.081222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.083786 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.083764    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-45127b63-serving-cert\""
Jul 14 00:10:11.091659 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.091627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:11.129130 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.129102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvnb\" (UniqueName: \"kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.129273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.129148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.129273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.129245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.230357 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.230320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvnb\" (UniqueName: \"kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.230541 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.230374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.230541 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.230442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.230659 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:10:11.230542    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-serving-cert: secret "maas-api-e2e-mt-45127b63-serving-cert" not found
Jul 14 00:10:11.230659 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:10:11.230604    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls podName:2654432b-9f49-4386-9e37-039f80b29703 nodeName:}" failed. No retries permitted until 2026-07-14 00:10:11.730583126 +0000 UTC m=+1860.017801451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls") pod "maas-api-e2e-mt-45127b63-58959bff89-wkzcm" (UID: "2654432b-9f49-4386-9e37-039f80b29703") : secret "maas-api-e2e-mt-45127b63-serving-cert" not found
Jul 14 00:10:11.231267 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.231247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.239290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.239264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvnb\" (UniqueName: \"kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.735674 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.735636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.738160 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.738119    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") pod \"maas-api-e2e-mt-45127b63-58959bff89-wkzcm\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:11.995668 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:11.995595    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:12.136006 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:12.135982    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:12.137617 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:10:12.137586    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2654432b_9f49_4386_9e37_039f80b29703.slice/crio-038a6a4fabec04743835e972498fff3e769a6f284a1e57c93a894313359b07e5 WatchSource:0}: Error finding container 038a6a4fabec04743835e972498fff3e769a6f284a1e57c93a894313359b07e5: Status 404 returned error can't find the container with id 038a6a4fabec04743835e972498fff3e769a6f284a1e57c93a894313359b07e5
Jul 14 00:10:12.877604 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:12.877516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" event={"ID":"2654432b-9f49-4386-9e37-039f80b29703","Type":"ContainerStarted","Data":"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"}
Jul 14 00:10:12.877604 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:12.877573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:12.877604 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:12.877599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" event={"ID":"2654432b-9f49-4386-9e37-039f80b29703","Type":"ContainerStarted","Data":"038a6a4fabec04743835e972498fff3e769a6f284a1e57c93a894313359b07e5"}
Jul 14 00:10:12.895499 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:12.895457    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" podStartSLOduration=1.420229683 podStartE2EDuration="1.895442526s" totalImagesPullingTime="475.212843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:10:12.893762529 +0000 UTC m=+1861.180980875" watchObservedRunningTime="2026-07-14 00:10:12.895442526 +0000 UTC m=+1861.182660871"
Jul 14 00:10:18.888577 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:18.888545    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:24.863820 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.863786    2558 patch_prober.go:28] interesting pod/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 14 00:10:24.864266 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.863849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 14 00:10:24.938515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.938484    2558 generic.go:356] "Generic (PLEG): container finished" podID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerID="0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e" exitCode=0
Jul 14 00:10:24.938658 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.938556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerDied","Data":"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"}
Jul 14 00:10:24.938936 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.938919    2558 scope.go:117] "RemoveContainer" containerID="0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"
Jul 14 00:10:24.939831 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.939805    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:24.939931 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.939874    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:24.939989 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:24.939929    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:10:25.952832 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:25.952791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerStarted","Data":"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"}
Jul 14 00:10:26.083800 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:26.083760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:26.083992 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:26.083975    2558 patch_prober.go:28] interesting pod/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 14 00:10:26.084064 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:26.084045    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 14 00:10:27.084228 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:27.084192    2558 patch_prober.go:28] interesting pod/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 14 00:10:27.084641 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:27.084251    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 14 00:10:28.084435 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:28.084400    2558 patch_prober.go:28] interesting pod/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 14 00:10:28.084841 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:28.084454    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 14 00:10:29.084917 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:29.084868    2558 patch_prober.go:28] interesting pod/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 14 00:10:29.085384 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:29.084964    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 14 00:10:30.088312 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:30.088275    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:30.088805 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:30.088655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:30.089386 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:30.089366    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:32.988723 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:32.988691    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:32.989492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:32.989449    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" podUID="2654432b-9f49-4386-9e37-039f80b29703" containerName="maas-api" containerID="cri-o://09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772" gracePeriod=30
Jul 14 00:10:33.241657 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.241607    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:33.331587 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.331561    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca\") pod \"2654432b-9f49-4386-9e37-039f80b29703\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") "
Jul 14 00:10:33.331741 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.331626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xvnb\" (UniqueName: \"kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb\") pod \"2654432b-9f49-4386-9e37-039f80b29703\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") "
Jul 14 00:10:33.331741 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.331698    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") pod \"2654432b-9f49-4386-9e37-039f80b29703\" (UID: \"2654432b-9f49-4386-9e37-039f80b29703\") "
Jul 14 00:10:33.331981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.331959    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2654432b-9f49-4386-9e37-039f80b29703" (UID: "2654432b-9f49-4386-9e37-039f80b29703"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:10:33.333633 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.333604    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb" (OuterVolumeSpecName: "kube-api-access-6xvnb") pod "2654432b-9f49-4386-9e37-039f80b29703" (UID: "2654432b-9f49-4386-9e37-039f80b29703"). InnerVolumeSpecName "kube-api-access-6xvnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:10:33.333728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.333696    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2654432b-9f49-4386-9e37-039f80b29703" (UID: "2654432b-9f49-4386-9e37-039f80b29703"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:10:33.432410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.432385    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2654432b-9f49-4386-9e37-039f80b29703-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:33.432410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.432407    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2654432b-9f49-4386-9e37-039f80b29703-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:33.432588 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.432417    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvnb\" (UniqueName: \"kubernetes.io/projected/2654432b-9f49-4386-9e37-039f80b29703-kube-api-access-6xvnb\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:33.996534 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.996460    2558 generic.go:356] "Generic (PLEG): container finished" podID="2654432b-9f49-4386-9e37-039f80b29703" containerID="09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772" exitCode=0
Jul 14 00:10:33.996534 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.996506    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"
Jul 14 00:10:33.996994 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.996544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" event={"ID":"2654432b-9f49-4386-9e37-039f80b29703","Type":"ContainerDied","Data":"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"}
Jul 14 00:10:33.996994 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.996597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm" event={"ID":"2654432b-9f49-4386-9e37-039f80b29703","Type":"ContainerDied","Data":"038a6a4fabec04743835e972498fff3e769a6f284a1e57c93a894313359b07e5"}
Jul 14 00:10:33.996994 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:33.996621    2558 scope.go:117] "RemoveContainer" containerID="09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"
Jul 14 00:10:34.005135 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:34.005120    2558 scope.go:117] "RemoveContainer" containerID="09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"
Jul 14 00:10:34.005404 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:10:34.005385    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772\": container with ID starting with 09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772 not found: ID does not exist" containerID="09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"
Jul 14 00:10:34.005480 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:34.005411    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772"} err="failed to get container status \"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772\": rpc error: code = NotFound desc = could not find container \"09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772\": container with ID starting with 09f4dffb789e21e4fcd0658861a9612b89ccd9a44ce7adfd5c5cc22403549772 not found: ID does not exist"
Jul 14 00:10:34.012252 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:34.012230    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:34.016108 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:34.016080    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45127b63-58959bff89-wkzcm"]
Jul 14 00:10:35.431497 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:35.431463    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2654432b-9f49-4386-9e37-039f80b29703" path="/var/lib/kubelet/pods/2654432b-9f49-4386-9e37-039f80b29703/volumes"
Jul 14 00:10:48.361503 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:48.361464    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:48.362041 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:48.361763    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy" containerID="cri-o://75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187" gracePeriod=30
Jul 14 00:10:53.619226 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.619200    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:53.699065 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.698992    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699065 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699124    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699165    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv6w9\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699186    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699215    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699280    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo\") pod \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\" (UID: \"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04\") "
Jul 14 00:10:53.699575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699418    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:10:53.699575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699450    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:10:53.699575 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:10:53.699754 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699609    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.699754 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699624    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:10:53.699754 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699629    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.699754 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.699664    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.700158 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.700132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data" (OuterVolumeSpecName: "istio-data") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:10:53.701412 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.701380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:10:53.701515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.701434    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9" (OuterVolumeSpecName: "kube-api-access-wv6w9") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "kube-api-access-wv6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:10:53.701570 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.701552    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:10:53.701633 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.701620    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token" (OuterVolumeSpecName: "istio-token") pod "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" (UID: "f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:10:53.800540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800512    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv6w9\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-kube-api-access-wv6w9\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.800540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800536    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.800540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800546    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.800716 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800554    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.800716 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800562    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:53.800716 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:53.800570    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:10:54.099299 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.099269    2558 generic.go:356] "Generic (PLEG): container finished" podID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerID="75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187" exitCode=0
Jul 14 00:10:54.099473 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.099318    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"
Jul 14 00:10:54.099473 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.099355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerDied","Data":"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"}
Jul 14 00:10:54.099473 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.099415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z" event={"ID":"f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04","Type":"ContainerDied","Data":"8379ba0c44c11603d50177949f44a3c9eb41f7bcfebe793e116f6e15b16c32cb"}
Jul 14 00:10:54.099473 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.099447    2558 scope.go:117] "RemoveContainer" containerID="75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"
Jul 14 00:10:54.109040 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.109022    2558 scope.go:117] "RemoveContainer" containerID="0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"
Jul 14 00:10:54.116693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.116676    2558 scope.go:117] "RemoveContainer" containerID="75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"
Jul 14 00:10:54.116939 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:10:54.116919    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187\": container with ID starting with 75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187 not found: ID does not exist" containerID="75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"
Jul 14 00:10:54.117021 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.116951    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187"} err="failed to get container status \"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187\": rpc error: code = NotFound desc = could not find container \"75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187\": container with ID starting with 75ca692e9e37aa31831c96e9a8c98fb87d551340c9fdf6ae323f04a480dcc187 not found: ID does not exist"
Jul 14 00:10:54.117021 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.116976    2558 scope.go:117] "RemoveContainer" containerID="0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"
Jul 14 00:10:54.117226 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:10:54.117205    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e\": container with ID starting with 0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e not found: ID does not exist" containerID="0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"
Jul 14 00:10:54.117265 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.117232    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e"} err="failed to get container status \"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e\": rpc error: code = NotFound desc = could not find container \"0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e\": container with ID starting with 0a33345870467a11dc661654ca41a9beb297df3d472c3741b98bcd035d54ea3e not found: ID does not exist"
Jul 14 00:10:54.120941 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.120920    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:54.124714 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:54.124695    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-45127b63-openshift-default-76dbffb7f9-zgl2z"]
Jul 14 00:10:55.432074 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:10:55.432046    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" path="/var/lib/kubelet/pods/f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04/volumes"
Jul 14 00:11:07.722781 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.722748    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723180    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723202    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723245    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2654432b-9f49-4386-9e37-039f80b29703" containerName="maas-api"
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723256    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2654432b-9f49-4386-9e37-039f80b29703" containerName="maas-api"
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723341    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:07.723352 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723352    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2654432b-9f49-4386-9e37-039f80b29703" containerName="maas-api"
Jul 14 00:11:07.723697 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.723361    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:07.727125 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.727105    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.730017 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.729993    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ca846028-openshift-default-dockercfg-gkkqq\""
Jul 14 00:11:07.742316 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.742293    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:11:07.817348 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817531 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817531 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz46\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817531 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817531 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817503    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.817769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.817761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918761 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz46\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.918979 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.918974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919359 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919472 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919472 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919438    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919574 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919574 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.919767 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.919739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.921253 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.921224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.922018 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.921998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.927056 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.927037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz46\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:07.927444 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:07.927429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token\") pod \"e2e-mt-ca846028-openshift-default-79dc998955-rnclk\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") " pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:08.039078 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:08.039041    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:08.179045 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:08.179019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:11:08.180853 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:11:08.180825    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2583bae5_89dc_4c2e_8417_7c902ea02a24.slice/crio-b0467fb82d4d4bab08b998ab45bd9fe6e2cb41b1db78fdf74dbde09ee5da555a WatchSource:0}: Error finding container b0467fb82d4d4bab08b998ab45bd9fe6e2cb41b1db78fdf74dbde09ee5da555a: Status 404 returned error can't find the container with id b0467fb82d4d4bab08b998ab45bd9fe6e2cb41b1db78fdf74dbde09ee5da555a
Jul 14 00:11:08.182865 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:08.182831    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:08.182947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:08.182914    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:08.182947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:08.182944    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:09.174794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:09.174761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerStarted","Data":"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"}
Jul 14 00:11:09.175185 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:09.174815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerStarted","Data":"b0467fb82d4d4bab08b998ab45bd9fe6e2cb41b1db78fdf74dbde09ee5da555a"}
Jul 14 00:11:09.195808 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:09.195767    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podStartSLOduration=2.195753 podStartE2EDuration="2.195753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:11:09.194064888 +0000 UTC m=+1917.481283233" watchObservedRunningTime="2026-07-14 00:11:09.195753 +0000 UTC m=+1917.482971347"
Jul 14 00:11:10.039812 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:10.039775    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:10.045808 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:10.045784    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:10.179636 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:10.179601    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:10.180572 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:10.180546    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:11.377720 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.377684    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:11.378326 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.378305    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:11.378397 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.378330    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f37ef9c9-33a5-4ce7-ad5e-9c21c31a4d04" containerName="istio-proxy"
Jul 14 00:11:11.380835 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.380811    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.383405 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.383381    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ca846028-serving-cert\""
Jul 14 00:11:11.392408 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.392385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:11.458241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.458215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76t5r\" (UniqueName: \"kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.458415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.458297    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.458415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.458322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.559761 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.559722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.559761 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.559761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.560047 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:11:11.559968    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-serving-cert: secret "maas-api-e2e-mt-ca846028-serving-cert" not found
Jul 14 00:11:11.560047 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:11:11.560034    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls podName:3faf1283-7574-41e0-8b99-d50d8a8fd181 nodeName:}" failed. No retries permitted until 2026-07-14 00:11:12.060018446 +0000 UTC m=+1920.347236770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls") pod "maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" (UID: "3faf1283-7574-41e0-8b99-d50d8a8fd181") : secret "maas-api-e2e-mt-ca846028-serving-cert" not found
Jul 14 00:11:11.560198 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.560100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76t5r\" (UniqueName: \"kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.560498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.560474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:11.570246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:11.570223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76t5r\" (UniqueName: \"kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:12.065688 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:12.065651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:12.065870 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:11:12.065799    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-serving-cert: secret "maas-api-e2e-mt-ca846028-serving-cert" not found
Jul 14 00:11:12.065959 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:11:12.065906    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls podName:3faf1283-7574-41e0-8b99-d50d8a8fd181 nodeName:}" failed. No retries permitted until 2026-07-14 00:11:13.065865534 +0000 UTC m=+1921.353083858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls") pod "maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" (UID: "3faf1283-7574-41e0-8b99-d50d8a8fd181") : secret "maas-api-e2e-mt-ca846028-serving-cert" not found
Jul 14 00:11:13.074077 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:13.074042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:13.076567 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:13.076544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"maas-api-e2e-mt-ca846028-869bd857ff-hpsg7\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:13.193309 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:13.193287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:13.325075 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:13.325007    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:13.326186 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:11:13.326156    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faf1283_7574_41e0_8b99_d50d8a8fd181.slice/crio-5666cbe6f65235fa87337a6a67d5c232508cecbbde0e57e8a020515476bd58d8 WatchSource:0}: Error finding container 5666cbe6f65235fa87337a6a67d5c232508cecbbde0e57e8a020515476bd58d8: Status 404 returned error can't find the container with id 5666cbe6f65235fa87337a6a67d5c232508cecbbde0e57e8a020515476bd58d8
Jul 14 00:11:13.328006 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:13.327983    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:11:14.201988 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:14.201904    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" event={"ID":"3faf1283-7574-41e0-8b99-d50d8a8fd181","Type":"ContainerStarted","Data":"afe28363230b32d088b1f9bee5a15cf8c6a7d59105f1773e9bf1e74976498468"}
Jul 14 00:11:14.201988 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:14.201958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" event={"ID":"3faf1283-7574-41e0-8b99-d50d8a8fd181","Type":"ContainerStarted","Data":"5666cbe6f65235fa87337a6a67d5c232508cecbbde0e57e8a020515476bd58d8"}
Jul 14 00:11:14.202360 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:14.201991    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:14.218826 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:14.218783    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" podStartSLOduration=2.754023473 podStartE2EDuration="3.218766725s" totalImagesPullingTime="464.743252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:11:14.217301469 +0000 UTC m=+1922.504519814" watchObservedRunningTime="2026-07-14 00:11:14.218766725 +0000 UTC m=+1922.505985070"
Jul 14 00:11:20.212135 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:20.212104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:22.346598 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:22.346564    2558 scope.go:117] "RemoveContainer" containerID="92ef0a7e2b625e908c3e6f391922c6348713785a74da131dae06a26795a1b645"
Jul 14 00:11:25.566055 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:25.565934    2558 patch_prober.go:28] interesting pod/e2e-mt-ca846028-openshift-default-79dc998955-rnclk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40920->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Jul 14 00:11:25.566055 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:25.566041    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:40920->10.133.0.58:15021: read: connection reset by peer"
Jul 14 00:11:26.270675 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.270636    2558 generic.go:356] "Generic (PLEG): container finished" podID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerID="043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729" exitCode=0
Jul 14 00:11:26.270857 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.270714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerDied","Data":"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"}
Jul 14 00:11:26.271127 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.271113    2558 scope.go:117] "RemoveContainer" containerID="043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"
Jul 14 00:11:26.272059 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.272029    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:26.272168 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.272110    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:26.272168 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:26.272146    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:11:27.278131 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:27.278099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerStarted","Data":"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"}
Jul 14 00:11:28.040002 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:28.039966    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:28.040423 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:28.040391    2558 patch_prober.go:28] interesting pod/e2e-mt-ca846028-openshift-default-79dc998955-rnclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 14 00:11:28.040529 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:28.040481    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 14 00:11:29.039846 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:29.039809    2558 patch_prober.go:28] interesting pod/e2e-mt-ca846028-openshift-default-79dc998955-rnclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 14 00:11:29.040452 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:29.040420    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 14 00:11:30.039836 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:30.039794    2558 patch_prober.go:28] interesting pod/e2e-mt-ca846028-openshift-default-79dc998955-rnclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 14 00:11:30.040160 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:30.040124    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 14 00:11:31.039481 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:31.039437    2558 patch_prober.go:28] interesting pod/e2e-mt-ca846028-openshift-default-79dc998955-rnclk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 14 00:11:31.039662 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:31.039554    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 14 00:11:32.042756 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:32.042719    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:32.043145 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:32.043005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:32.043678 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:32.043658    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:11:43.736029 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.735989    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"]
Jul 14 00:11:43.738671 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.738655    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:43.741110 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.741089    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:11:43.747610 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.747587    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"]
Jul 14 00:11:43.853635 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.853607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca846028-zxhwh\" (UID: \"eb74759e-b3ac-40db-85ad-63d318248c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:43.954707 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.954672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca846028-zxhwh\" (UID: \"eb74759e-b3ac-40db-85ad-63d318248c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:43.955294 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:43.955275    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca846028-zxhwh\" (UID: \"eb74759e-b3ac-40db-85ad-63d318248c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:44.051625 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:44.051598    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:44.186724 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:44.186698    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"]
Jul 14 00:11:44.188860 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:11:44.188830    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb74759e_b3ac_40db_85ad_63d318248c3e.slice/crio-ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba WatchSource:0}: Error finding container ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba: Status 404 returned error can't find the container with id ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba
Jul 14 00:11:44.363128 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:44.363094    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh" event={"ID":"eb74759e-b3ac-40db-85ad-63d318248c3e","Type":"ContainerStarted","Data":"ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba"}
Jul 14 00:11:45.369819 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:45.369727    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb74759e-b3ac-40db-85ad-63d318248c3e" containerID="67aecf6cc002e02c16ddd288f8be36008afb3d09d43f0b6bf4392fbff99d1335" exitCode=0
Jul 14 00:11:45.369819 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:45.369768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh" event={"ID":"eb74759e-b3ac-40db-85ad-63d318248c3e","Type":"ContainerDied","Data":"67aecf6cc002e02c16ddd288f8be36008afb3d09d43f0b6bf4392fbff99d1335"}
Jul 14 00:11:46.506619 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:46.506596    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:46.575383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:46.575355    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca\") pod \"eb74759e-b3ac-40db-85ad-63d318248c3e\" (UID: \"eb74759e-b3ac-40db-85ad-63d318248c3e\") "
Jul 14 00:11:46.575732 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:46.575713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb74759e-b3ac-40db-85ad-63d318248c3e" (UID: "eb74759e-b3ac-40db-85ad-63d318248c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:11:46.676674 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:46.676611    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb74759e-b3ac-40db-85ad-63d318248c3e-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:11:47.380661 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:47.380636    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh"
Jul 14 00:11:47.380838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:47.380686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca846028-zxhwh" event={"ID":"eb74759e-b3ac-40db-85ad-63d318248c3e","Type":"ContainerDied","Data":"ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba"}
Jul 14 00:11:47.380838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:47.380732    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff52a3568947e685d571d0ade760df95d0fe4f6f7012a63b8835e294e8c422ba"
Jul 14 00:11:54.193190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.193157    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:54.193673 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.193385    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" podUID="3faf1283-7574-41e0-8b99-d50d8a8fd181" containerName="maas-api" containerID="cri-o://afe28363230b32d088b1f9bee5a15cf8c6a7d59105f1773e9bf1e74976498468" gracePeriod=30
Jul 14 00:11:54.414654 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.414624    2558 generic.go:356] "Generic (PLEG): container finished" podID="3faf1283-7574-41e0-8b99-d50d8a8fd181" containerID="afe28363230b32d088b1f9bee5a15cf8c6a7d59105f1773e9bf1e74976498468" exitCode=0
Jul 14 00:11:54.414795 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.414672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" event={"ID":"3faf1283-7574-41e0-8b99-d50d8a8fd181","Type":"ContainerDied","Data":"afe28363230b32d088b1f9bee5a15cf8c6a7d59105f1773e9bf1e74976498468"}
Jul 14 00:11:54.450110 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.450049    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:54.539313 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.539286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76t5r\" (UniqueName: \"kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r\") pod \"3faf1283-7574-41e0-8b99-d50d8a8fd181\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") "
Jul 14 00:11:54.539443 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.539318    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") pod \"3faf1283-7574-41e0-8b99-d50d8a8fd181\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") "
Jul 14 00:11:54.539443 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.539379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca\") pod \"3faf1283-7574-41e0-8b99-d50d8a8fd181\" (UID: \"3faf1283-7574-41e0-8b99-d50d8a8fd181\") "
Jul 14 00:11:54.539801 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.539776    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3faf1283-7574-41e0-8b99-d50d8a8fd181" (UID: "3faf1283-7574-41e0-8b99-d50d8a8fd181"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:11:54.541367 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.541334    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r" (OuterVolumeSpecName: "kube-api-access-76t5r") pod "3faf1283-7574-41e0-8b99-d50d8a8fd181" (UID: "3faf1283-7574-41e0-8b99-d50d8a8fd181"). InnerVolumeSpecName "kube-api-access-76t5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:11:54.541448 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.541342    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3faf1283-7574-41e0-8b99-d50d8a8fd181" (UID: "3faf1283-7574-41e0-8b99-d50d8a8fd181"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:11:54.640377 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.640355    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3faf1283-7574-41e0-8b99-d50d8a8fd181-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:11:54.640377 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.640377    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76t5r\" (UniqueName: \"kubernetes.io/projected/3faf1283-7574-41e0-8b99-d50d8a8fd181-kube-api-access-76t5r\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:11:54.640534 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:54.640386    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3faf1283-7574-41e0-8b99-d50d8a8fd181-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:11:55.428959 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:55.428923    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"
Jul 14 00:11:55.436308 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:55.436274    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7" event={"ID":"3faf1283-7574-41e0-8b99-d50d8a8fd181","Type":"ContainerDied","Data":"5666cbe6f65235fa87337a6a67d5c232508cecbbde0e57e8a020515476bd58d8"}
Jul 14 00:11:55.436451 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:55.436335    2558 scope.go:117] "RemoveContainer" containerID="afe28363230b32d088b1f9bee5a15cf8c6a7d59105f1773e9bf1e74976498468"
Jul 14 00:11:55.453896 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:55.453860    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:55.456984 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:55.456960    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ca846028-869bd857ff-hpsg7"]
Jul 14 00:11:57.432336 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:11:57.432305    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3faf1283-7574-41e0-8b99-d50d8a8fd181" path="/var/lib/kubelet/pods/3faf1283-7574-41e0-8b99-d50d8a8fd181/volumes"
Jul 14 00:12:51.065595 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:51.065554    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:12:51.066220 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:51.065776    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy" containerID="cri-o://54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a" gracePeriod=30
Jul 14 00:12:56.339349 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.339273    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:12:56.471188 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471154    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471188 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471195    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471238    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsz46\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471348    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471374    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471427    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471470 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471454    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs\") pod \"2583bae5-89dc-4c2e-8417-7c902ea02a24\" (UID: \"2583bae5-89dc-4c2e-8417-7c902ea02a24\") "
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471621    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471729    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471749    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.471838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.471773    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:12:56.472460 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.472435    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data" (OuterVolumeSpecName: "istio-data") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:12:56.473564 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.473532    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token" (OuterVolumeSpecName: "istio-token") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:12:56.473799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.473592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46" (OuterVolumeSpecName: "kube-api-access-gsz46") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "kube-api-access-gsz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:12:56.473799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.473656    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:12:56.473799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.473713    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2583bae5-89dc-4c2e-8417-7c902ea02a24" (UID: "2583bae5-89dc-4c2e-8417-7c902ea02a24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:12:56.572252 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572218    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572252 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572247    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572252 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572257    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572483 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572265    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz46\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-kube-api-access-gsz46\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572483 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572275    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2583bae5-89dc-4c2e-8417-7c902ea02a24-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572483 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572282    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2583bae5-89dc-4c2e-8417-7c902ea02a24-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.572483 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.572290    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2583bae5-89dc-4c2e-8417-7c902ea02a24-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:12:56.735105 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.735012    2558 generic.go:356] "Generic (PLEG): container finished" podID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerID="54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a" exitCode=0
Jul 14 00:12:56.735105 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.735073    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"
Jul 14 00:12:56.735344 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.735099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerDied","Data":"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"}
Jul 14 00:12:56.735344 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.735167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk" event={"ID":"2583bae5-89dc-4c2e-8417-7c902ea02a24","Type":"ContainerDied","Data":"b0467fb82d4d4bab08b998ab45bd9fe6e2cb41b1db78fdf74dbde09ee5da555a"}
Jul 14 00:12:56.735344 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.735197    2558 scope.go:117] "RemoveContainer" containerID="54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"
Jul 14 00:12:56.744796 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.744778    2558 scope.go:117] "RemoveContainer" containerID="043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"
Jul 14 00:12:56.752636 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.752617    2558 scope.go:117] "RemoveContainer" containerID="54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"
Jul 14 00:12:56.752924 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:12:56.752902    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a\": container with ID starting with 54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a not found: ID does not exist" containerID="54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"
Jul 14 00:12:56.753002 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.752931    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a"} err="failed to get container status \"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a\": rpc error: code = NotFound desc = could not find container \"54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a\": container with ID starting with 54bab666e046ac07ae45f0ab9c375d6f6d5089b3b5de5b282b525373af6b934a not found: ID does not exist"
Jul 14 00:12:56.753002 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.752947    2558 scope.go:117] "RemoveContainer" containerID="043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"
Jul 14 00:12:56.753225 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:12:56.753205    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729\": container with ID starting with 043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729 not found: ID does not exist" containerID="043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"
Jul 14 00:12:56.753293 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.753229    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729"} err="failed to get container status \"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729\": rpc error: code = NotFound desc = could not find container \"043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729\": container with ID starting with 043e8f473fcf5dd77432fd57ad78866d53255d3bcca536624bb1641366561729 not found: ID does not exist"
Jul 14 00:12:56.758341 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.758307    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:12:56.760538 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:56.760520    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ca846028-openshift-default-79dc998955-rnclk"]
Jul 14 00:12:57.432533 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:12:57.432502    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" path="/var/lib/kubelet/pods/2583bae5-89dc-4c2e-8417-7c902ea02a24/volumes"
Jul 14 00:13:45.343129 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343044    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:13:45.343578 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343561    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:45.343621 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343581    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:45.343621 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343597    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3faf1283-7574-41e0-8b99-d50d8a8fd181" containerName="maas-api"
Jul 14 00:13:45.343621 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343605    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3faf1283-7574-41e0-8b99-d50d8a8fd181" containerName="maas-api"
Jul 14 00:13:45.343621 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343619    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:45.343621 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343628    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:45.343794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343651    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb74759e-b3ac-40db-85ad-63d318248c3e" containerName="revoke-keys"
Jul 14 00:13:45.343794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343661    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb74759e-b3ac-40db-85ad-63d318248c3e" containerName="revoke-keys"
Jul 14 00:13:45.343794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343748    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb74759e-b3ac-40db-85ad-63d318248c3e" containerName="revoke-keys"
Jul 14 00:13:45.343794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343758    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:45.343794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.343772    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3faf1283-7574-41e0-8b99-d50d8a8fd181" containerName="maas-api"
Jul 14 00:13:45.348385 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.348366    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.351142 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.351125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-36055a-openshift-default-dockercfg-w46b9\""
Jul 14 00:13:45.356776 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.356752    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:13:45.474397 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474627 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474627 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474627 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474627 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474627 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474956 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cx4\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474956 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.474956 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.474773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575578 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.575728 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576062 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.575978    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576062 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576189 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576189 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cx4\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576189 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576355 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.576492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.576476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.578101 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.578078    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.578310 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.578288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.584000 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.583978    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.584225 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.584205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cx4\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4\") pod \"e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") " pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.660272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.660204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:45.797219 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.797193    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:13:45.798283 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:13:45.798252    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74f0566_0b5c_4fbc_a1ce_c4a9147cf9ec.slice/crio-b1338c0724e4399d7ef65399ff6540a4a89205ef63a472e3a5b9e893b38a8023 WatchSource:0}: Error finding container b1338c0724e4399d7ef65399ff6540a4a89205ef63a472e3a5b9e893b38a8023: Status 404 returned error can't find the container with id b1338c0724e4399d7ef65399ff6540a4a89205ef63a472e3a5b9e893b38a8023
Jul 14 00:13:45.800415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.800383    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:13:45.800492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.800442    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:13:45.800492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.800473    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:13:45.969070 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.968982    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerStarted","Data":"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"}
Jul 14 00:13:45.969070 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.969047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerStarted","Data":"b1338c0724e4399d7ef65399ff6540a4a89205ef63a472e3a5b9e893b38a8023"}
Jul 14 00:13:45.990023 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:45.989975    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podStartSLOduration=0.989960182 podStartE2EDuration="989.960182ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:13:45.987569851 +0000 UTC m=+2074.274788196" watchObservedRunningTime="2026-07-14 00:13:45.989960182 +0000 UTC m=+2074.277178528"
Jul 14 00:13:46.660402 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:46.660368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:46.665239 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:46.665211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:46.973877 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:46.973785    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:46.974595 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:46.974576    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:13:48.010865 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.010825    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:13:48.011504 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.011487    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2583bae5-89dc-4c2e-8417-7c902ea02a24" containerName="istio-proxy"
Jul 14 00:13:48.014592 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.014572    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.017215 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.017191    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-36055a-serving-cert\""
Jul 14 00:13:48.023746 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.023721    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:13:48.100229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.100195    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.100365 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.100296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.100365 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.100360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk422\" (UniqueName: \"kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.201608 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.201571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.201784 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.201622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.201784 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.201650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk422\" (UniqueName: \"kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.202233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.202201    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.204229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.204195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.210116 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.210097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk422\" (UniqueName: \"kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422\") pod \"maas-api-e2e-inf-a-36055a-5468957cd5-r64fq\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.328079 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.328002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:48.459587 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.459554    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:13:48.460263 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:13:48.460227    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fac88e_d6f5_4bba_bf01_cb059651ee7e.slice/crio-a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57 WatchSource:0}: Error finding container a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57: Status 404 returned error can't find the container with id a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57
Jul 14 00:13:48.983654 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:48.983614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" event={"ID":"89fac88e-d6f5-4bba-bf01-cb059651ee7e","Type":"ContainerStarted","Data":"a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57"}
Jul 14 00:13:49.990200 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:49.990165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" event={"ID":"89fac88e-d6f5-4bba-bf01-cb059651ee7e","Type":"ContainerStarted","Data":"7bd68dbab81e0d552a63b10c2758883979435c7102e48ee9100aba5dd6c89349"}
Jul 14 00:13:49.990630 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:49.990232    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:50.011579 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:50.011531    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" podStartSLOduration=2.4620774389999998 podStartE2EDuration="3.011518477s" totalImagesPullingTime="549.441038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:13:50.00891099 +0000 UTC m=+2078.296129335" watchObservedRunningTime="2026-07-14 00:13:50.011518477 +0000 UTC m=+2078.298736823"
Jul 14 00:13:55.686233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.686196    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:13:55.690133 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.690115    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.692721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.692698    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-00e043-serving-cert\""
Jul 14 00:13:55.702634 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.702611    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:13:55.771043 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.771016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.771199 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.771054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8278\" (UniqueName: \"kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.771199 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.771089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.872405 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.872363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.872584 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:13:55.872514    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-serving-cert: secret "maas-api-e2e-inf-b-00e043-serving-cert" not found
Jul 14 00:13:55.872584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.872523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8278\" (UniqueName: \"kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.872723 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:13:55.872589    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls podName:70795ce4-89fe-4995-b748-f3caaca77c42 nodeName:}" failed. No retries permitted until 2026-07-14 00:13:56.372568209 +0000 UTC m=+2084.659786550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls") pod "maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" (UID: "70795ce4-89fe-4995-b748-f3caaca77c42") : secret "maas-api-e2e-inf-b-00e043-serving-cert" not found
Jul 14 00:13:55.872723 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.872644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.873257 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.873239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:55.881581 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:55.881557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8278\" (UniqueName: \"kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:56.000266 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:56.000182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:13:56.377036 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:56.377007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:56.379505 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:56.379477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") pod \"maas-api-e2e-inf-b-00e043-5bff749958-9mkbz\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:56.603468 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:56.603440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:56.760818 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:56.760476    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:13:56.767792 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:13:56.763731    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70795ce4_89fe_4995_b748_f3caaca77c42.slice/crio-16593a7715987bc829186d3aa9c1ed32eeeeb94e56f9d3781cfc39c5cf8c6348 WatchSource:0}: Error finding container 16593a7715987bc829186d3aa9c1ed32eeeeb94e56f9d3781cfc39c5cf8c6348: Status 404 returned error can't find the container with id 16593a7715987bc829186d3aa9c1ed32eeeeb94e56f9d3781cfc39c5cf8c6348
Jul 14 00:13:57.031137 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:57.031092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" event={"ID":"70795ce4-89fe-4995-b748-f3caaca77c42","Type":"ContainerStarted","Data":"16593a7715987bc829186d3aa9c1ed32eeeeb94e56f9d3781cfc39c5cf8c6348"}
Jul 14 00:13:58.044290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:58.044249    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" event={"ID":"70795ce4-89fe-4995-b748-f3caaca77c42","Type":"ContainerStarted","Data":"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"}
Jul 14 00:13:58.044937 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:58.044917    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:13:58.065585 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:13:58.065534    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" podStartSLOduration=2.395526829 podStartE2EDuration="3.065515118s" totalImagesPullingTime="669.988289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:13:58.061973531 +0000 UTC m=+2086.349191877" watchObservedRunningTime="2026-07-14 00:13:58.065515118 +0000 UTC m=+2086.352733465"
Jul 14 00:14:03.015110 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:03.015059    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 00:14:03.015507 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:03.015145    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 00:14:04.077156 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.077123    2558 generic.go:356] "Generic (PLEG): container finished" podID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerID="25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb" exitCode=0
Jul 14 00:14:04.077573 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.077172    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerDied","Data":"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"}
Jul 14 00:14:04.077573 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.077520    2558 scope.go:117] "RemoveContainer" containerID="25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"
Jul 14 00:14:04.078569 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.078535    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:14:04.078643 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.078625    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:14:04.078687 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:04.078673    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:14:05.062320 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:05.062289    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:14:05.087450 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:05.087395    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerStarted","Data":"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"}
Jul 14 00:14:05.660614 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:05.660576    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:14:05.660806 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:05.660785    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 00:14:05.660851 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:05.660824    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 00:14:06.661450 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:06.661407    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 00:14:06.661949 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:06.661468    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 00:14:07.660871 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:07.660838    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 00:14:07.661075 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:07.660914    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 00:14:08.661487 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:08.661450    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 00:14:08.661856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:08.661507    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 00:14:09.663983 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:09.663952    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:14:09.664380 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:09.664284    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:14:09.664819 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:09.664799    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:14:22.396947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:22.396915    2558 scope.go:117] "RemoveContainer" containerID="41c74872ce413c09925710e0c7a202527010b235d1ea30f028ccce6d30e2887c"
Jul 14 00:14:44.356922 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.356871    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"]
Jul 14 00:14:44.360233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.360216    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:44.362560 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.362536    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:14:44.367427 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.367400    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"]
Jul 14 00:14:44.507492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.507464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn\" (UID: \"904e4e3c-b8ae-42e8-a00a-fe4544e22834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:44.608261 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.608189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn\" (UID: \"904e4e3c-b8ae-42e8-a00a-fe4544e22834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:44.608713 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.608697    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn\" (UID: \"904e4e3c-b8ae-42e8-a00a-fe4544e22834\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:44.672248 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.672224    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:44.806783 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:44.806760    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"]
Jul 14 00:14:44.808907 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:14:44.808864    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904e4e3c_b8ae_42e8_a00a_fe4544e22834.slice/crio-e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4 WatchSource:0}: Error finding container e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4: Status 404 returned error can't find the container with id e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4
Jul 14 00:14:45.296402 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:45.296298    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn" event={"ID":"904e4e3c-b8ae-42e8-a00a-fe4544e22834","Type":"ContainerStarted","Data":"c7bdf6677aa53a49b425d9d5b1e589bde8a6a7b145b59c7d6042f0b35cb28b65"}
Jul 14 00:14:45.296546 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:45.296453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn" event={"ID":"904e4e3c-b8ae-42e8-a00a-fe4544e22834","Type":"ContainerStarted","Data":"e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4"}
Jul 14 00:14:45.312650 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:45.312606    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn" podStartSLOduration=1.312592985 podStartE2EDuration="1.312592985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:14:45.310353275 +0000 UTC m=+2133.597571623" watchObservedRunningTime="2026-07-14 00:14:45.312592985 +0000 UTC m=+2133.599811331"
Jul 14 00:14:46.303464 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:46.303422    2558 generic.go:356] "Generic (PLEG): container finished" podID="904e4e3c-b8ae-42e8-a00a-fe4544e22834" containerID="c7bdf6677aa53a49b425d9d5b1e589bde8a6a7b145b59c7d6042f0b35cb28b65" exitCode=0
Jul 14 00:14:46.303841 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:46.303482    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn" event={"ID":"904e4e3c-b8ae-42e8-a00a-fe4544e22834","Type":"ContainerDied","Data":"c7bdf6677aa53a49b425d9d5b1e589bde8a6a7b145b59c7d6042f0b35cb28b65"}
Jul 14 00:14:47.442658 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:47.442635    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:47.531916 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:47.531870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca\") pod \"904e4e3c-b8ae-42e8-a00a-fe4544e22834\" (UID: \"904e4e3c-b8ae-42e8-a00a-fe4544e22834\") "
Jul 14 00:14:47.532233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:47.532210    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "904e4e3c-b8ae-42e8-a00a-fe4544e22834" (UID: "904e4e3c-b8ae-42e8-a00a-fe4544e22834"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:14:47.632868 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:47.632801    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904e4e3c-b8ae-42e8-a00a-fe4544e22834-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:14:48.313963 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:48.313922    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn" event={"ID":"904e4e3c-b8ae-42e8-a00a-fe4544e22834","Type":"ContainerDied","Data":"e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4"}
Jul 14 00:14:48.314167 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:48.313988    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e378770488caab3ac93131d006b8203476ea239efc7c291aef7ec2d2dfd620a4"
Jul 14 00:14:48.314167 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:48.313942    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn"
Jul 14 00:14:54.903846 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:54.903810    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:14:54.904320 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:54.904098    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" podUID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" containerName="maas-api" containerID="cri-o://7bd68dbab81e0d552a63b10c2758883979435c7102e48ee9100aba5dd6c89349" gracePeriod=30
Jul 14 00:14:55.352245 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.352219    2558 generic.go:356] "Generic (PLEG): container finished" podID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" containerID="7bd68dbab81e0d552a63b10c2758883979435c7102e48ee9100aba5dd6c89349" exitCode=0
Jul 14 00:14:55.352387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.352268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" event={"ID":"89fac88e-d6f5-4bba-bf01-cb059651ee7e","Type":"ContainerDied","Data":"7bd68dbab81e0d552a63b10c2758883979435c7102e48ee9100aba5dd6c89349"}
Jul 14 00:14:55.352387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.352315    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq" event={"ID":"89fac88e-d6f5-4bba-bf01-cb059651ee7e","Type":"ContainerDied","Data":"a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57"}
Jul 14 00:14:55.352387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.352341    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a110d8f68cf65f34b6c00ff6878dc65e75c67986ee3856bd8d92d2e2f10b1f57"
Jul 14 00:14:55.354455 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.354430    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:14:55.409256 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.409234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca\") pod \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") "
Jul 14 00:14:55.409385 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.409277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk422\" (UniqueName: \"kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422\") pod \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") "
Jul 14 00:14:55.409434 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.409381    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls\") pod \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\" (UID: \"89fac88e-d6f5-4bba-bf01-cb059651ee7e\") "
Jul 14 00:14:55.409624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.409602    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89fac88e-d6f5-4bba-bf01-cb059651ee7e" (UID: "89fac88e-d6f5-4bba-bf01-cb059651ee7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:14:55.409709 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.409662    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fac88e-d6f5-4bba-bf01-cb059651ee7e-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:14:55.411342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.411318    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422" (OuterVolumeSpecName: "kube-api-access-dk422") pod "89fac88e-d6f5-4bba-bf01-cb059651ee7e" (UID: "89fac88e-d6f5-4bba-bf01-cb059651ee7e"). InnerVolumeSpecName "kube-api-access-dk422". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:14:55.411471 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.411455    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89fac88e-d6f5-4bba-bf01-cb059651ee7e" (UID: "89fac88e-d6f5-4bba-bf01-cb059651ee7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:14:55.510976 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.510950    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk422\" (UniqueName: \"kubernetes.io/projected/89fac88e-d6f5-4bba-bf01-cb059651ee7e-kube-api-access-dk422\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:14:55.511074 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:55.510977    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fac88e-d6f5-4bba-bf01-cb059651ee7e-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:14:56.357012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:56.356977    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"
Jul 14 00:14:56.373317 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:56.373278    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:14:56.375193 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:56.375174    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36055a-5468957cd5-r64fq"]
Jul 14 00:14:57.432491 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:14:57.432457    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" path="/var/lib/kubelet/pods/89fac88e-d6f5-4bba-bf01-cb059651ee7e/volumes"
Jul 14 00:15:00.141844 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.141812    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"]
Jul 14 00:15:00.142273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142243    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904e4e3c-b8ae-42e8-a00a-fe4544e22834" containerName="revoke-keys"
Jul 14 00:15:00.142273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142259    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="904e4e3c-b8ae-42e8-a00a-fe4544e22834" containerName="revoke-keys"
Jul 14 00:15:00.142273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142271    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" containerName="maas-api"
Jul 14 00:15:00.142375 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142277    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" containerName="maas-api"
Jul 14 00:15:00.142375 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142356    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="89fac88e-d6f5-4bba-bf01-cb059651ee7e" containerName="maas-api"
Jul 14 00:15:00.142375 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.142368    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="904e4e3c-b8ae-42e8-a00a-fe4544e22834" containerName="revoke-keys"
Jul 14 00:15:00.144635 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.144616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"]
Jul 14 00:15:00.144753 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.144733    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"
Jul 14 00:15:00.146942 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.146927    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"
Jul 14 00:15:00.147403 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.147388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:15:00.154057 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.154032    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"]
Jul 14 00:15:00.156342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.156321    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"]
Jul 14 00:15:00.156521 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.156507    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"
Jul 14 00:15:00.165252 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.165231    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"
Jul 14 00:15:00.309950 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.309699    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"]
Jul 14 00:15:00.311410 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:15:00.311372    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c71533c_f1fa_4765_a3b8_fe7b3d8d2270.slice/crio-c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356 WatchSource:0}: Error finding container c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356: Status 404 returned error can't find the container with id c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356
Jul 14 00:15:00.333079 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.333010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"]
Jul 14 00:15:00.335143 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:15:00.335119    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b984051_4368_4bdc_9d79_6483bbe12618.slice/crio-c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160 WatchSource:0}: Error finding container c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160: Status 404 returned error can't find the container with id c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160
Jul 14 00:15:00.379580 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.379552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g" event={"ID":"9b984051-4368-4bdc-9d79-6483bbe12618","Type":"ContainerStarted","Data":"c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160"}
Jul 14 00:15:00.380616 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:00.380596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6" event={"ID":"3c71533c-f1fa-4765-a3b8-fe7b3d8d2270","Type":"ContainerStarted","Data":"c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356"}
Jul 14 00:15:01.386525 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:01.386485    2558 generic.go:356] "Generic (PLEG): container finished" podID="9b984051-4368-4bdc-9d79-6483bbe12618" containerID="789fb298544b95d673f9a4317bcd8a3de7661405bdca10b9eb1ca963f047e736" exitCode=0
Jul 14 00:15:01.386951 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:01.386572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g" event={"ID":"9b984051-4368-4bdc-9d79-6483bbe12618","Type":"ContainerDied","Data":"789fb298544b95d673f9a4317bcd8a3de7661405bdca10b9eb1ca963f047e736"}
Jul 14 00:15:01.388100 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:01.388068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6" event={"ID":"3c71533c-f1fa-4765-a3b8-fe7b3d8d2270","Type":"ContainerStarted","Data":"f1d91eaed752f61a5bc1e4e3cbd22ce5ea7f734c2ce7cdc7f7bc5640bbcc6d70"}
Jul 14 00:15:01.416316 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:01.416258    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6" podStartSLOduration=1.416240722 podStartE2EDuration="1.416240722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:15:01.415010858 +0000 UTC m=+2149.702229201" watchObservedRunningTime="2026-07-14 00:15:01.416240722 +0000 UTC m=+2149.703459068"
Jul 14 00:15:02.394175 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:02.394138    2558 generic.go:356] "Generic (PLEG): container finished" podID="3c71533c-f1fa-4765-a3b8-fe7b3d8d2270" containerID="f1d91eaed752f61a5bc1e4e3cbd22ce5ea7f734c2ce7cdc7f7bc5640bbcc6d70" exitCode=0
Jul 14 00:15:02.394631 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:02.394223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6" event={"ID":"3c71533c-f1fa-4765-a3b8-fe7b3d8d2270","Type":"ContainerDied","Data":"f1d91eaed752f61a5bc1e4e3cbd22ce5ea7f734c2ce7cdc7f7bc5640bbcc6d70"}
Jul 14 00:15:02.525952 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:02.525869    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"
Jul 14 00:15:03.400950 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:03.400923    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g"
Jul 14 00:15:03.401334 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:03.400949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733135-blz7g" event={"ID":"9b984051-4368-4bdc-9d79-6483bbe12618","Type":"ContainerDied","Data":"c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160"}
Jul 14 00:15:03.401334 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:03.400992    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89200f07c59455a6bd5673246451c91c636d1372c3783dc45aa784496805160"
Jul 14 00:15:03.536585 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:03.536561    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"
Jul 14 00:15:04.407415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:04.407345    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"
Jul 14 00:15:04.407415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:04.407354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6" event={"ID":"3c71533c-f1fa-4765-a3b8-fe7b3d8d2270","Type":"ContainerDied","Data":"c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356"}
Jul 14 00:15:04.407415 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:04.407400    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c28c37363a66ccc23e41184eddc64710b14287e16620056109f73b9135076356"
Jul 14 00:15:10.110042 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.110011    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:15:10.110434 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.110210    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy" containerID="cri-o://751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486" gracePeriod=30
Jul 14 00:15:10.483940 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.483851    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"]
Jul 14 00:15:10.484241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484228    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b984051-4368-4bdc-9d79-6483bbe12618" containerName="cleanup"
Jul 14 00:15:10.484292 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484244    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b984051-4368-4bdc-9d79-6483bbe12618" containerName="cleanup"
Jul 14 00:15:10.484292 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484258    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c71533c-f1fa-4765-a3b8-fe7b3d8d2270" containerName="cleanup"
Jul 14 00:15:10.484292 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484264    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c71533c-f1fa-4765-a3b8-fe7b3d8d2270" containerName="cleanup"
Jul 14 00:15:10.484386 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484331    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c71533c-f1fa-4765-a3b8-fe7b3d8d2270" containerName="cleanup"
Jul 14 00:15:10.484386 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.484340    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b984051-4368-4bdc-9d79-6483bbe12618" containerName="cleanup"
Jul 14 00:15:10.486357 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.486342    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:10.489098 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.489077    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:15:10.495202 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.495175    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"]
Jul 14 00:15:10.544780 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.544750    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00e043-9qz49\" (UID: \"7ee8ef46-e860-4760-b037-23151e9b0e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:10.645624 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.645592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00e043-9qz49\" (UID: \"7ee8ef46-e860-4760-b037-23151e9b0e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:10.646234 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.646214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00e043-9qz49\" (UID: \"7ee8ef46-e860-4760-b037-23151e9b0e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:10.798043 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.798019    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:10.933545 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:10.933521    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"]
Jul 14 00:15:10.936414 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:15:10.936377    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee8ef46_e860_4760_b037_23151e9b0e89.slice/crio-1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e WatchSource:0}: Error finding container 1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e: Status 404 returned error can't find the container with id 1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e
Jul 14 00:15:11.444588 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:11.444486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49" event={"ID":"7ee8ef46-e860-4760-b037-23151e9b0e89","Type":"ContainerStarted","Data":"67c2641f2569a244c6fb292066043e9172c7104caac35cdd999fca52c521525a"}
Jul 14 00:15:11.444588 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:11.444545    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49" event={"ID":"7ee8ef46-e860-4760-b037-23151e9b0e89","Type":"ContainerStarted","Data":"1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e"}
Jul 14 00:15:11.460185 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:11.460141    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49" podStartSLOduration=1.460126823 podStartE2EDuration="1.460126823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:15:11.458587696 +0000 UTC m=+2159.745806042" watchObservedRunningTime="2026-07-14 00:15:11.460126823 +0000 UTC m=+2159.747345169"
Jul 14 00:15:12.453074 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:12.453040    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ee8ef46-e860-4760-b037-23151e9b0e89" containerID="67c2641f2569a244c6fb292066043e9172c7104caac35cdd999fca52c521525a" exitCode=0
Jul 14 00:15:12.453422 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:12.453133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49" event={"ID":"7ee8ef46-e860-4760-b037-23151e9b0e89","Type":"ContainerDied","Data":"67c2641f2569a244c6fb292066043e9172c7104caac35cdd999fca52c521525a"}
Jul 14 00:15:13.586934 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:13.586912    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:13.671941 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:13.671915    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca\") pod \"7ee8ef46-e860-4760-b037-23151e9b0e89\" (UID: \"7ee8ef46-e860-4760-b037-23151e9b0e89\") "
Jul 14 00:15:13.672273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:13.672252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ee8ef46-e860-4760-b037-23151e9b0e89" (UID: "7ee8ef46-e860-4760-b037-23151e9b0e89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:15:13.773396 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:13.773371    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee8ef46-e860-4760-b037-23151e9b0e89-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:14.464311 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:14.464284    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49"
Jul 14 00:15:14.464515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:14.464322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00e043-9qz49" event={"ID":"7ee8ef46-e860-4760-b037-23151e9b0e89","Type":"ContainerDied","Data":"1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e"}
Jul 14 00:15:14.464515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:14.464364    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0a59f53f4368b8b553b2cbf71edee4e91fbe666afda45550f07f8ec608fd3e"
Jul 14 00:15:15.354020 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.353981    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:15:15.470791 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.470718    2558 generic.go:356] "Generic (PLEG): container finished" podID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerID="751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486" exitCode=0
Jul 14 00:15:15.470791 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.470761    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"
Jul 14 00:15:15.471003 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.470799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerDied","Data":"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"}
Jul 14 00:15:15.471003 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.470849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq" event={"ID":"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec","Type":"ContainerDied","Data":"b1338c0724e4399d7ef65399ff6540a4a89205ef63a472e3a5b9e893b38a8023"}
Jul 14 00:15:15.471003 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.470875    2558 scope.go:117] "RemoveContainer" containerID="751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"
Jul 14 00:15:15.479407 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.479392    2558 scope.go:117] "RemoveContainer" containerID="25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"
Jul 14 00:15:15.487168 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.487152    2558 scope.go:117] "RemoveContainer" containerID="751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"
Jul 14 00:15:15.487407 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:15.487388    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486\": container with ID starting with 751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486 not found: ID does not exist" containerID="751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"
Jul 14 00:15:15.487451 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.487421    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486"} err="failed to get container status \"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486\": rpc error: code = NotFound desc = could not find container \"751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486\": container with ID starting with 751a35ad576a3195fb2d5a3198987169d60700b8680ddcf1cb0ea6a7b45e4486 not found: ID does not exist"
Jul 14 00:15:15.487451 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.487439    2558 scope.go:117] "RemoveContainer" containerID="25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"
Jul 14 00:15:15.487671 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:15.487654    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb\": container with ID starting with 25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb not found: ID does not exist" containerID="25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"
Jul 14 00:15:15.487722 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.487675    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb"} err="failed to get container status \"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb\": rpc error: code = NotFound desc = could not find container \"25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb\": container with ID starting with 25f2b49590ae278e929f5eadb4b615658eb978d3c5196eaa9835a9ecc8a7fecb not found: ID does not exist"
Jul 14 00:15:15.489951 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.489935    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490022 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.489963    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490022 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.489996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490022 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490050    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490082    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490168    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:15:15.490229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8cx4\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4\") pod \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\" (UID: \"c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec\") "
Jul 14 00:15:15.490492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:15:15.490597 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:15:15.490665 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490621    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.490665 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490643    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.490769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.490696    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:15:15.491317 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.491294    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:15:15.492290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.492209    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:15:15.492560 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.492536    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:15:15.492661 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.492587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4" (OuterVolumeSpecName: "kube-api-access-k8cx4") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "kube-api-access-k8cx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:15:15.492661 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.492608    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" (UID: "c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:15:15.591588 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591567    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591588 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591588    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8cx4\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-kube-api-access-k8cx4\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591597    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591606    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591615    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591623    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.591721 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.591630    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:15.792268 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.792239    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:15:15.796616 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:15.796591    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-36055a-openshift-default-5db8cbf985-grwmq"]
Jul 14 00:15:17.437404 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:17.437365    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" path="/var/lib/kubelet/pods/c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec/volumes"
Jul 14 00:15:21.090691 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.090654    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:15:21.091131 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.090926    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" podUID="70795ce4-89fe-4995-b748-f3caaca77c42" containerName="maas-api" containerID="cri-o://cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1" gracePeriod=30
Jul 14 00:15:21.117215 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.117186    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"]
Jul 14 00:15:21.121153 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.121129    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-00e043-29733135-2tsn6"]
Jul 14 00:15:21.346827 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.346775    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:15:21.432739 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.432703    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c71533c-f1fa-4765-a3b8-fe7b3d8d2270" path="/var/lib/kubelet/pods/3c71533c-f1fa-4765-a3b8-fe7b3d8d2270/volumes"
Jul 14 00:15:21.440799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.440776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca\") pod \"70795ce4-89fe-4995-b748-f3caaca77c42\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") "
Jul 14 00:15:21.440949 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.440805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8278\" (UniqueName: \"kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278\") pod \"70795ce4-89fe-4995-b748-f3caaca77c42\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") "
Jul 14 00:15:21.440949 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.440833    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") pod \"70795ce4-89fe-4995-b748-f3caaca77c42\" (UID: \"70795ce4-89fe-4995-b748-f3caaca77c42\") "
Jul 14 00:15:21.441172 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.441146    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70795ce4-89fe-4995-b748-f3caaca77c42" (UID: "70795ce4-89fe-4995-b748-f3caaca77c42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:15:21.442872 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.442845    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70795ce4-89fe-4995-b748-f3caaca77c42" (UID: "70795ce4-89fe-4995-b748-f3caaca77c42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:15:21.443052 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.442879    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278" (OuterVolumeSpecName: "kube-api-access-v8278") pod "70795ce4-89fe-4995-b748-f3caaca77c42" (UID: "70795ce4-89fe-4995-b748-f3caaca77c42"). InnerVolumeSpecName "kube-api-access-v8278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:15:21.504433 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.504406    2558 generic.go:356] "Generic (PLEG): container finished" podID="70795ce4-89fe-4995-b748-f3caaca77c42" containerID="cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1" exitCode=0
Jul 14 00:15:21.504519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.504447    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"
Jul 14 00:15:21.504519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.504464    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" event={"ID":"70795ce4-89fe-4995-b748-f3caaca77c42","Type":"ContainerDied","Data":"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"}
Jul 14 00:15:21.504519 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.504502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz" event={"ID":"70795ce4-89fe-4995-b748-f3caaca77c42","Type":"ContainerDied","Data":"16593a7715987bc829186d3aa9c1ed32eeeeb94e56f9d3781cfc39c5cf8c6348"}
Jul 14 00:15:21.504646 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.504522    2558 scope.go:117] "RemoveContainer" containerID="cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"
Jul 14 00:15:21.513540 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.513520    2558 scope.go:117] "RemoveContainer" containerID="cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"
Jul 14 00:15:21.513800 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:21.513783    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1\": container with ID starting with cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1 not found: ID does not exist" containerID="cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"
Jul 14 00:15:21.513862 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.513810    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1"} err="failed to get container status \"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1\": rpc error: code = NotFound desc = could not find container \"cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1\": container with ID starting with cf401b05faa7b15bbd5203a2cdd2e91e3ff93ba5620e8a2f52b0dc4a2c97c0e1 not found: ID does not exist"
Jul 14 00:15:21.526578 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.526556    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:15:21.530393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.530370    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00e043-5bff749958-9mkbz"]
Jul 14 00:15:21.541614 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.541591    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70795ce4-89fe-4995-b748-f3caaca77c42-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:21.541709 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.541616    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8278\" (UniqueName: \"kubernetes.io/projected/70795ce4-89fe-4995-b748-f3caaca77c42-kube-api-access-v8278\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:21.541709 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:21.541628    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70795ce4-89fe-4995-b748-f3caaca77c42-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:15:23.432697 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:23.432661    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70795ce4-89fe-4995-b748-f3caaca77c42" path="/var/lib/kubelet/pods/70795ce4-89fe-4995-b748-f3caaca77c42/volumes"
Jul 14 00:15:37.131497 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131464    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131808    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131819    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131827    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee8ef46-e860-4760-b037-23151e9b0e89" containerName="revoke-keys"
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131833    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee8ef46-e860-4760-b037-23151e9b0e89" containerName="revoke-keys"
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131846    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70795ce4-89fe-4995-b748-f3caaca77c42" containerName="maas-api"
Jul 14 00:15:37.131855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131852    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="70795ce4-89fe-4995-b748-f3caaca77c42" containerName="maas-api"
Jul 14 00:15:37.132232 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131933    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:37.132232 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131943    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:37.132232 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131956    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="70795ce4-89fe-4995-b748-f3caaca77c42" containerName="maas-api"
Jul 14 00:15:37.132232 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.131966    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee8ef46-e860-4760-b037-23151e9b0e89" containerName="revoke-keys"
Jul 14 00:15:37.137801 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.137779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.140199 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.140178    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-211a7b-openshift-default-dockercfg-4kh7t\""
Jul 14 00:15:37.150092 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.150062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:15:37.179847 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrhbh\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.179981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.179981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.179981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.180183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179983    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.180183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.179999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.180183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.180035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.180183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.180054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.180183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.180073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281277 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281277 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281848 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281542    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrhbh\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281848 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281848 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281734    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.281848 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281815    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.282094 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.281997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.282159 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.282115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.283647 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.283625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.284015 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.283997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.289188 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.289166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.289508 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.289491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrhbh\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh\") pod \"e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") " pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.457793 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.457708    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:37.627552 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.627527    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:15:37.629046 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:15:37.629011    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad15295_227e_4599_8c00_14c5585b892a.slice/crio-a552240a653b67e96dcb55f061733e27b2e2a3279cc582e68889b43755088d89 WatchSource:0}: Error finding container a552240a653b67e96dcb55f061733e27b2e2a3279cc582e68889b43755088d89: Status 404 returned error can't find the container with id a552240a653b67e96dcb55f061733e27b2e2a3279cc582e68889b43755088d89
Jul 14 00:15:37.631229 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.631197    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:37.631297 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.631271    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:37.631335 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:37.631304    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:38.590340 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:38.590306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerStarted","Data":"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"}
Jul 14 00:15:38.590718 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:38.590356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerStarted","Data":"a552240a653b67e96dcb55f061733e27b2e2a3279cc582e68889b43755088d89"}
Jul 14 00:15:38.611083 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:38.611045    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podStartSLOduration=1.611031058 podStartE2EDuration="1.611031058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:15:38.608736998 +0000 UTC m=+2186.895955347" watchObservedRunningTime="2026-07-14 00:15:38.611031058 +0000 UTC m=+2186.898249404"
Jul 14 00:15:39.458299 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:39.458267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:39.463672 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:39.463618    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:39.595282 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:39.595242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:39.596281 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:39.596263    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:40.659995 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.659947    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:15:40.660583 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.660561    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:40.660651 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.660586    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f0566-0b5c-4fbc-a1ce-c4a9147cf9ec" containerName="istio-proxy"
Jul 14 00:15:40.664144 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.664128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.666666 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.666646    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-211a7b-serving-cert\""
Jul 14 00:15:40.673658 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.673637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:15:40.711775 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.711732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.711929 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.711794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4m97\" (UniqueName: \"kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.712069 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.712048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.813279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.813250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.813431 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.813298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.813431 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.813333    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4m97\" (UniqueName: \"kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.813431 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:40.813408    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-serving-cert: secret "maas-api-e2e-api-a-211a7b-serving-cert" not found
Jul 14 00:15:40.813552 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:40.813500    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls podName:3ca6b14f-abea-4d07-8f08-72a67f5d400b nodeName:}" failed. No retries permitted until 2026-07-14 00:15:41.31347887 +0000 UTC m=+2189.600697196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls") pod "maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" (UID: "3ca6b14f-abea-4d07-8f08-72a67f5d400b") : secret "maas-api-e2e-api-a-211a7b-serving-cert" not found
Jul 14 00:15:40.814011 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.813993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:40.825243 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:40.825221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4m97\" (UniqueName: \"kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:41.319363 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:41.319329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:41.321742 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:41.321708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") pod \"maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:41.578012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:41.577939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:41.717636 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:41.717614    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:15:41.719149 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:15:41.719123    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca6b14f_abea_4d07_8f08_72a67f5d400b.slice/crio-b64ea21a0e34eb993356b8984cdb90e2d4675ff259f364fdc7408ddc9480f377 WatchSource:0}: Error finding container b64ea21a0e34eb993356b8984cdb90e2d4675ff259f364fdc7408ddc9480f377: Status 404 returned error can't find the container with id b64ea21a0e34eb993356b8984cdb90e2d4675ff259f364fdc7408ddc9480f377
Jul 14 00:15:42.617740 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:42.617710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" event={"ID":"3ca6b14f-abea-4d07-8f08-72a67f5d400b","Type":"ContainerStarted","Data":"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"}
Jul 14 00:15:42.617867 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:42.617773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" event={"ID":"3ca6b14f-abea-4d07-8f08-72a67f5d400b","Type":"ContainerStarted","Data":"b64ea21a0e34eb993356b8984cdb90e2d4675ff259f364fdc7408ddc9480f377"}
Jul 14 00:15:42.617867 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:42.617812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:42.636310 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:42.636266    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" podStartSLOduration=1.844032737 podStartE2EDuration="2.636249102s" totalImagesPullingTime="792.216365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:15:42.634727986 +0000 UTC m=+2190.921946333" watchObservedRunningTime="2026-07-14 00:15:42.636249102 +0000 UTC m=+2190.923467447"
Jul 14 00:15:48.639167 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:48.639132    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:15:49.327511 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.322717    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:15:49.350115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.331935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.350115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.341086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:15:49.350115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.341227    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1ad924-serving-cert\""
Jul 14 00:15:49.392742 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.392716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzrf\" (UniqueName: \"kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.392916 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.392751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.392916 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.392788    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.493552 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.493512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzrf\" (UniqueName: \"kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.493552 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.493551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.493796 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.493589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.493796 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:49.493718    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-serving-cert: secret "maas-api-e2e-api-b-1ad924-serving-cert" not found
Jul 14 00:15:49.493796 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:15:49.493785    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls podName:3a02f1da-4a18-479e-a643-625eab04057a nodeName:}" failed. No retries permitted until 2026-07-14 00:15:49.993769166 +0000 UTC m=+2198.280987490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls") pod "maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" (UID: "3a02f1da-4a18-479e-a643-625eab04057a") : secret "maas-api-e2e-api-b-1ad924-serving-cert" not found
Jul 14 00:15:49.494233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.494215    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.505004 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.504983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzrf\" (UniqueName: \"kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:49.998625 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:49.998584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:50.001288 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:50.001263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") pod \"maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:50.266261 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:50.266233    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:50.409512 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:50.409480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:15:50.672555 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:50.672467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" event={"ID":"3a02f1da-4a18-479e-a643-625eab04057a","Type":"ContainerStarted","Data":"d02b3d5f070c294c838c2e5b5e7b64fda24b1a8d962934b7b828a8c67c6555d3"}
Jul 14 00:15:51.679051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:51.679007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" event={"ID":"3a02f1da-4a18-479e-a643-625eab04057a","Type":"ContainerStarted","Data":"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"}
Jul 14 00:15:51.679425 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:51.679101    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:51.698050 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:51.698005    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" podStartSLOduration=2.100539598 podStartE2EDuration="2.697990818s" totalImagesPullingTime="597.45122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:15:51.694973356 +0000 UTC m=+2199.982191702" watchObservedRunningTime="2026-07-14 00:15:51.697990818 +0000 UTC m=+2199.985209164"
Jul 14 00:15:53.690149 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.690113    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ad15295-227e-4599-8c00-14c5585b892a" containerID="58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6" exitCode=0
Jul 14 00:15:53.690564 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.690180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerDied","Data":"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"}
Jul 14 00:15:53.690564 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.690546    2558 scope.go:117] "RemoveContainer" containerID="58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"
Jul 14 00:15:53.691516 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.691488    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:53.691620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.691566    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:53.691620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:53.691614    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:15:54.596176 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:54.596149    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:54.697138 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:54.697108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerStarted","Data":"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"}
Jul 14 00:15:55.458495 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:55.458462    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:55.459643 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:55.459617    2558 patch_prober.go:28] interesting pod/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 14 00:15:55.459759 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:55.459665    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 14 00:15:56.458946 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:56.458907    2558 patch_prober.go:28] interesting pod/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 14 00:15:56.459351 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:56.458972    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 14 00:15:57.459012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:57.458977    2558 patch_prober.go:28] interesting pod/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 14 00:15:57.459381 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:57.459032    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 14 00:15:57.689490 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:57.689459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:15:58.459026 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:58.458991    2558 patch_prober.go:28] interesting pod/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 14 00:15:58.459476 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:58.459046    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 14 00:15:59.461622 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:59.461588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:59.462008 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:59.461945    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:15:59.462565 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:15:59.462539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:16:12.357844 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.357800    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:16:12.358312 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.358082    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" podUID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" containerName="maas-api" containerID="cri-o://526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315" gracePeriod=30
Jul 14 00:16:12.605723 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.605701    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:16:12.707149 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.707077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4m97\" (UniqueName: \"kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97\") pod \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") "
Jul 14 00:16:12.707149 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.707121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca\") pod \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") "
Jul 14 00:16:12.707314 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.707156    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") pod \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\" (UID: \"3ca6b14f-abea-4d07-8f08-72a67f5d400b\") "
Jul 14 00:16:12.707483 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.707452    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ca6b14f-abea-4d07-8f08-72a67f5d400b" (UID: "3ca6b14f-abea-4d07-8f08-72a67f5d400b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:16:12.709184 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.709153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97" (OuterVolumeSpecName: "kube-api-access-d4m97") pod "3ca6b14f-abea-4d07-8f08-72a67f5d400b" (UID: "3ca6b14f-abea-4d07-8f08-72a67f5d400b"). InnerVolumeSpecName "kube-api-access-d4m97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:16:12.709271 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.709221    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ca6b14f-abea-4d07-8f08-72a67f5d400b" (UID: "3ca6b14f-abea-4d07-8f08-72a67f5d400b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:16:12.793246 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.793217    2558 generic.go:356] "Generic (PLEG): container finished" podID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" containerID="526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315" exitCode=0
Jul 14 00:16:12.793342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.793258    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"
Jul 14 00:16:12.793342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.793270    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" event={"ID":"3ca6b14f-abea-4d07-8f08-72a67f5d400b","Type":"ContainerDied","Data":"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"}
Jul 14 00:16:12.793342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.793309    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh" event={"ID":"3ca6b14f-abea-4d07-8f08-72a67f5d400b","Type":"ContainerDied","Data":"b64ea21a0e34eb993356b8984cdb90e2d4675ff259f364fdc7408ddc9480f377"}
Jul 14 00:16:12.793342 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.793331    2558 scope.go:117] "RemoveContainer" containerID="526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"
Jul 14 00:16:12.801929 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.801909    2558 scope.go:117] "RemoveContainer" containerID="526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"
Jul 14 00:16:12.802170 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:12.802148    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315\": container with ID starting with 526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315 not found: ID does not exist" containerID="526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"
Jul 14 00:16:12.802226 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.802179    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315"} err="failed to get container status \"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315\": rpc error: code = NotFound desc = could not find container \"526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315\": container with ID starting with 526175d0235e9c1f8a07a0d9b02411132ebc71b40bd877b8ea500e181ee11315 not found: ID does not exist"
Jul 14 00:16:12.808691 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.808674    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6b14f-abea-4d07-8f08-72a67f5d400b-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:12.808757 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.808692    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca6b14f-abea-4d07-8f08-72a67f5d400b-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:12.808757 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.808702    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4m97\" (UniqueName: \"kubernetes.io/projected/3ca6b14f-abea-4d07-8f08-72a67f5d400b-kube-api-access-d4m97\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:12.814726 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.814705    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:16:12.818617 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:12.818596    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-211a7b-646f86d58b-sr2hh"]
Jul 14 00:16:13.433927 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:13.433877    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" path="/var/lib/kubelet/pods/3ca6b14f-abea-4d07-8f08-72a67f5d400b/volumes"
Jul 14 00:16:27.362359 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.362324    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:16:27.362877 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.362619    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" containerID="cri-o://c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11" gracePeriod=30
Jul 14 00:16:27.728498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.728399    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"]
Jul 14 00:16:27.728972 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.728954    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" containerName="maas-api"
Jul 14 00:16:27.729027 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.728975    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" containerName="maas-api"
Jul 14 00:16:27.729100 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.729089    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca6b14f-abea-4d07-8f08-72a67f5d400b" containerName="maas-api"
Jul 14 00:16:27.732274 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.732248    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:27.735359 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.735334    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:16:27.738773 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.738744    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"]
Jul 14 00:16:27.834387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.834347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z\" (UID: \"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:27.935535 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.935495    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z\" (UID: \"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:27.936163 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:27.936141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z\" (UID: \"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:28.046665 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:28.046634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:28.179381 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:28.179360    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"]
Jul 14 00:16:28.182104 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:16:28.182074    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ef65f3_bb2a_46fc_bf56_4ce1a38e2ec9.slice/crio-92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9 WatchSource:0}: Error finding container 92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9: Status 404 returned error can't find the container with id 92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9
Jul 14 00:16:28.877006 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:28.876971    2558 generic.go:356] "Generic (PLEG): container finished" podID="83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9" containerID="1fc574f5e1a270cb33305a46b56cf03554a7708c2dfef280c09f6e8a7fb7092a" exitCode=0
Jul 14 00:16:28.877332 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:28.877012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z" event={"ID":"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9","Type":"ContainerDied","Data":"1fc574f5e1a270cb33305a46b56cf03554a7708c2dfef280c09f6e8a7fb7092a"}
Jul 14 00:16:28.877332 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:28.877052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z" event={"ID":"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9","Type":"ContainerStarted","Data":"92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9"}
Jul 14 00:16:29.464218 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:29.464176    2558 patch_prober.go:28] interesting pod/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 00:16:29.464374 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:29.464239    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 00:16:30.014507 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.014482    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:30.052484 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.052449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca\") pod \"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9\" (UID: \"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9\") "
Jul 14 00:16:30.052826 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.052795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9" (UID: "83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:16:30.153819 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.153752    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:30.888584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.888551    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z" event={"ID":"83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9","Type":"ContainerDied","Data":"92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9"}
Jul 14 00:16:30.888584 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.888582    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z"
Jul 14 00:16:30.888807 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:30.888601    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92fc5ba39d8f6b500a022b1627002c8f7fdf631a1f3d1a8ecd9b56fc70a994b9"
Jul 14 00:16:32.603983 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.603959    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:16:32.674058 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674058 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674058 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674051    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrhbh\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674173    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674237    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674261    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert\") pod \"9ad15295-227e-4599-8c00-14c5585b892a\" (UID: \"9ad15295-227e-4599-8c00-14c5585b892a\") "
Jul 14 00:16:32.674579 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:16:32.674579 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:16:32.674579 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:16:32.674719 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674627    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.674719 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674648    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.674719 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674650    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:16:32.674719 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.674665    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.675338 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.675314    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:16:32.676393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.676367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:16:32.676492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.676422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:16:32.676492 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.676458    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh" (OuterVolumeSpecName: "kube-api-access-lrhbh") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "kube-api-access-lrhbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:16:32.676683 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.676667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ad15295-227e-4599-8c00-14c5585b892a" (UID: "9ad15295-227e-4599-8c00-14c5585b892a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:16:32.775084 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775060    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.775084 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775081    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad15295-227e-4599-8c00-14c5585b892a-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.775227 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775090    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrhbh\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-kube-api-access-lrhbh\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.775227 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775101    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad15295-227e-4599-8c00-14c5585b892a-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.775227 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775110    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad15295-227e-4599-8c00-14c5585b892a-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.775227 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.775118    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad15295-227e-4599-8c00-14c5585b892a-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:32.901280 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.901245    2558 generic.go:356] "Generic (PLEG): container finished" podID="9ad15295-227e-4599-8c00-14c5585b892a" containerID="c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11" exitCode=0
Jul 14 00:16:32.901410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.901297    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"
Jul 14 00:16:32.901410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.901341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerDied","Data":"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"}
Jul 14 00:16:32.901410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.901394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m" event={"ID":"9ad15295-227e-4599-8c00-14c5585b892a","Type":"ContainerDied","Data":"a552240a653b67e96dcb55f061733e27b2e2a3279cc582e68889b43755088d89"}
Jul 14 00:16:32.901518 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.901418    2558 scope.go:117] "RemoveContainer" containerID="c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"
Jul 14 00:16:32.910368 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.910342    2558 scope.go:117] "RemoveContainer" containerID="58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"
Jul 14 00:16:32.918383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.918369    2558 scope.go:117] "RemoveContainer" containerID="c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"
Jul 14 00:16:32.918629 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:32.918612    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11\": container with ID starting with c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11 not found: ID does not exist" containerID="c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"
Jul 14 00:16:32.918678 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.918637    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11"} err="failed to get container status \"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11\": rpc error: code = NotFound desc = could not find container \"c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11\": container with ID starting with c48ddcccb0a428feb09d4fbb4abb3e43c2813054004163e867a1ad4de69a3d11 not found: ID does not exist"
Jul 14 00:16:32.918678 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.918653    2558 scope.go:117] "RemoveContainer" containerID="58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"
Jul 14 00:16:32.918864 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:32.918849    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6\": container with ID starting with 58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6 not found: ID does not exist" containerID="58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"
Jul 14 00:16:32.918930 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.918867    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6"} err="failed to get container status \"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6\": rpc error: code = NotFound desc = could not find container \"58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6\": container with ID starting with 58ecbb0c462150a07c1e6cf0612207515bb65373b3d58a03e157d299b74f8ee6 not found: ID does not exist"
Jul 14 00:16:32.927981 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.927919    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:16:32.930033 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:32.930010    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-211a7b-openshift-default-6489ff477f-ghc9m"]
Jul 14 00:16:33.433024 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:33.432985    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad15295-227e-4599-8c00-14c5585b892a" path="/var/lib/kubelet/pods/9ad15295-227e-4599-8c00-14c5585b892a/volumes"
Jul 14 00:16:38.135183 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.135101    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:16:38.135613 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.135495    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" podUID="3a02f1da-4a18-479e-a643-625eab04057a" containerName="maas-api" containerID="cri-o://1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d" gracePeriod=30
Jul 14 00:16:38.384079 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.384056    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:16:38.419273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.419194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca\") pod \"3a02f1da-4a18-479e-a643-625eab04057a\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") "
Jul 14 00:16:38.419408 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.419279    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwzrf\" (UniqueName: \"kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf\") pod \"3a02f1da-4a18-479e-a643-625eab04057a\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") "
Jul 14 00:16:38.419408 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.419329    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") pod \"3a02f1da-4a18-479e-a643-625eab04057a\" (UID: \"3a02f1da-4a18-479e-a643-625eab04057a\") "
Jul 14 00:16:38.419547 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.419523    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a02f1da-4a18-479e-a643-625eab04057a" (UID: "3a02f1da-4a18-479e-a643-625eab04057a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:16:38.421431 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.421401    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf" (OuterVolumeSpecName: "kube-api-access-vwzrf") pod "3a02f1da-4a18-479e-a643-625eab04057a" (UID: "3a02f1da-4a18-479e-a643-625eab04057a"). InnerVolumeSpecName "kube-api-access-vwzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:16:38.421555 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.421436    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a02f1da-4a18-479e-a643-625eab04057a" (UID: "3a02f1da-4a18-479e-a643-625eab04057a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:16:38.520177 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.520146    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzrf\" (UniqueName: \"kubernetes.io/projected/3a02f1da-4a18-479e-a643-625eab04057a-kube-api-access-vwzrf\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:38.520177 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.520172    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a02f1da-4a18-479e-a643-625eab04057a-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:38.520177 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.520181    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a02f1da-4a18-479e-a643-625eab04057a-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:16:38.934460 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.934427    2558 generic.go:356] "Generic (PLEG): container finished" podID="3a02f1da-4a18-479e-a643-625eab04057a" containerID="1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d" exitCode=0
Jul 14 00:16:38.934663 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.934480    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"
Jul 14 00:16:38.934663 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.934494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" event={"ID":"3a02f1da-4a18-479e-a643-625eab04057a","Type":"ContainerDied","Data":"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"}
Jul 14 00:16:38.934663 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.934532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7" event={"ID":"3a02f1da-4a18-479e-a643-625eab04057a","Type":"ContainerDied","Data":"d02b3d5f070c294c838c2e5b5e7b64fda24b1a8d962934b7b828a8c67c6555d3"}
Jul 14 00:16:38.934663 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.934551    2558 scope.go:117] "RemoveContainer" containerID="1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"
Jul 14 00:16:38.943393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.943378    2558 scope.go:117] "RemoveContainer" containerID="1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"
Jul 14 00:16:38.943623 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:38.943603    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d\": container with ID starting with 1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d not found: ID does not exist" containerID="1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"
Jul 14 00:16:38.943663 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.943633    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d"} err="failed to get container status \"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d\": rpc error: code = NotFound desc = could not find container \"1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d\": container with ID starting with 1f000eb81f4d1f2607a11e94fa0b931761e020d3332f9a73d9107e500eb9290d not found: ID does not exist"
Jul 14 00:16:38.954905 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.954869    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:16:38.958428 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:38.958407    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ad924-796bd595b4-gwfh7"]
Jul 14 00:16:39.432746 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:39.432715    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a02f1da-4a18-479e-a643-625eab04057a" path="/var/lib/kubelet/pods/3a02f1da-4a18-479e-a643-625eab04057a/volumes"
Jul 14 00:16:54.365354 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365312    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:16:54.365856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365814    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9" containerName="revoke-keys"
Jul 14 00:16:54.365856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365833    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9" containerName="revoke-keys"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365870    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365879    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365918    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a02f1da-4a18-479e-a643-625eab04057a" containerName="maas-api"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365929    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a02f1da-4a18-479e-a643-625eab04057a" containerName="maas-api"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365941    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.365947    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.366023    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9" containerName="revoke-keys"
Jul 14 00:16:54.366272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.366033    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.366039    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad15295-227e-4599-8c00-14c5585b892a" containerName="istio-proxy"
Jul 14 00:16:54.366272 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.366045    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a02f1da-4a18-479e-a643-625eab04057a" containerName="maas-api"
Jul 14 00:16:54.368860 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.368843    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.371280 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.371258    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7e917f-openshift-default-dockercfg-9jzwq\""
Jul 14 00:16:54.380372 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.380354    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:16:54.448278 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448410 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448323    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448618 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t5h\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448618 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448618 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448618 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448618 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.448915 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.448629    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549461 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549461 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t5h\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549693 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549930 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549930 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.549930 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549848    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.550099 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.549927    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.550155 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.550117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.550249 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.550227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.550325 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.550312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.550368 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.550324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.552409 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.552390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.552539 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.552522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.557938 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.557913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.558136 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.558116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t5h\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h\") pod \"e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") " pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.679880 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.679811    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:54.815515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.815491    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:16:54.818212 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:16:54.818180    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bc8f41_d941_4d84_bf0e_4e8d0487cc3f.slice/crio-59941c3f8f3596b606847be3c25cba7fb0889af015d9ae51bd56886821a1d51f WatchSource:0}: Error finding container 59941c3f8f3596b606847be3c25cba7fb0889af015d9ae51bd56886821a1d51f: Status 404 returned error can't find the container with id 59941c3f8f3596b606847be3c25cba7fb0889af015d9ae51bd56886821a1d51f
Jul 14 00:16:54.820742 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.820702    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:16:54.820831 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.820789    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:16:54.820871 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:54.820830    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:16:55.018570 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:55.018537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerStarted","Data":"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"}
Jul 14 00:16:55.018730 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:55.018589    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerStarted","Data":"59941c3f8f3596b606847be3c25cba7fb0889af015d9ae51bd56886821a1d51f"}
Jul 14 00:16:55.038338 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:55.038288    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podStartSLOduration=1.0382684 podStartE2EDuration="1.0382684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:16:55.037374665 +0000 UTC m=+2263.324593013" watchObservedRunningTime="2026-07-14 00:16:55.0382684 +0000 UTC m=+2263.325486748"
Jul 14 00:16:55.680115 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:55.680074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:55.684916 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:55.684870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:56.024166 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:56.024129    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:56.025138 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:56.025111    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:16:57.192173 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.192140    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:16:57.205069 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.205039    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.207527 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.207505    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7e917f-serving-cert\""
Jul 14 00:16:57.209070 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.209043    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:16:57.274947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.274910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.275085 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.275007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krlc\" (UniqueName: \"kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.275085 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.275044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.376326 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.376292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.376513 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.376350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.376513 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.376442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krlc\" (UniqueName: \"kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.376608 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:57.376592    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-serving-cert: secret "maas-api-e2e-shared-a-7e917f-serving-cert" not found
Jul 14 00:16:57.376678 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:16:57.376667    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls podName:595318e8-457b-49fa-8bbb-53889c9ddf3a nodeName:}" failed. No retries permitted until 2026-07-14 00:16:57.87664935 +0000 UTC m=+2266.163867673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls") pod "maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" (UID: "595318e8-457b-49fa-8bbb-53889c9ddf3a") : secret "maas-api-e2e-shared-a-7e917f-serving-cert" not found
Jul 14 00:16:57.376970 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.376954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.387400 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.387378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krlc\" (UniqueName: \"kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.880048 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.880015    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:57.882445 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:57.882416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") pod \"maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:58.139745 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:58.139661    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:58.274725 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:58.274691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:16:58.276352 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:16:58.276329    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595318e8_457b_49fa_8bbb_53889c9ddf3a.slice/crio-02f8f75b68c9dad8a253af6dbe18b5ca8d504e7a70324cc702c04df2120f0505 WatchSource:0}: Error finding container 02f8f75b68c9dad8a253af6dbe18b5ca8d504e7a70324cc702c04df2120f0505: Status 404 returned error can't find the container with id 02f8f75b68c9dad8a253af6dbe18b5ca8d504e7a70324cc702c04df2120f0505
Jul 14 00:16:58.277703 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:58.277682    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:16:59.041737 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:59.041698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" event={"ID":"595318e8-457b-49fa-8bbb-53889c9ddf3a","Type":"ContainerStarted","Data":"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"}
Jul 14 00:16:59.041935 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:59.041757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" event={"ID":"595318e8-457b-49fa-8bbb-53889c9ddf3a","Type":"ContainerStarted","Data":"02f8f75b68c9dad8a253af6dbe18b5ca8d504e7a70324cc702c04df2120f0505"}
Jul 14 00:16:59.041935 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:59.041863    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:16:59.061305 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:16:59.061257    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" podStartSLOduration=1.507602757 podStartE2EDuration="2.061241044s" totalImagesPullingTime="553.638287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:16:59.059205219 +0000 UTC m=+2267.346423565" watchObservedRunningTime="2026-07-14 00:16:59.061241044 +0000 UTC m=+2267.348459389"
Jul 14 00:17:05.053586 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.053558    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:17:05.609742 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.609705    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:17:05.613820 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.613798    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.616474 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.616455    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-09117f-serving-cert\""
Jul 14 00:17:05.634267 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.634239    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:17:05.757043 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.757010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.757233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.757061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.757233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.757086    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xd4n\" (UniqueName: \"kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.857993 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.857959    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.858164 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.858004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.858164 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.858040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xd4n\" (UniqueName: \"kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.858284 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:17:05.858207    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-serving-cert: secret "maas-api-e2e-shared-b-09117f-serving-cert" not found
Jul 14 00:17:05.858346 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:17:05.858287    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls podName:c1f9ed61-63d0-4b17-93c5-2891166be705 nodeName:}" failed. No retries permitted until 2026-07-14 00:17:06.358264416 +0000 UTC m=+2274.645482756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls") pod "maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" (UID: "c1f9ed61-63d0-4b17-93c5-2891166be705") : secret "maas-api-e2e-shared-b-09117f-serving-cert" not found
Jul 14 00:17:05.858652 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.858632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:05.866940 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:05.866867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xd4n\" (UniqueName: \"kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:06.363344 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:06.363305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:06.365995 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:06.365967    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") pod \"maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:06.545008 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:06.544969    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:06.712355 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:06.712330    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:17:06.714045 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:17:06.714014    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f9ed61_63d0_4b17_93c5_2891166be705.slice/crio-923f95ec2435e2d1eefc100d1d3e3c1171a8c50cb53fee7c8ced41a736531df3 WatchSource:0}: Error finding container 923f95ec2435e2d1eefc100d1d3e3c1171a8c50cb53fee7c8ced41a736531df3: Status 404 returned error can't find the container with id 923f95ec2435e2d1eefc100d1d3e3c1171a8c50cb53fee7c8ced41a736531df3
Jul 14 00:17:07.083762 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:07.083727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" event={"ID":"c1f9ed61-63d0-4b17-93c5-2891166be705","Type":"ContainerStarted","Data":"923f95ec2435e2d1eefc100d1d3e3c1171a8c50cb53fee7c8ced41a736531df3"}
Jul 14 00:17:08.090224 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:08.090190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" event={"ID":"c1f9ed61-63d0-4b17-93c5-2891166be705","Type":"ContainerStarted","Data":"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"}
Jul 14 00:17:08.090613 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:08.090316    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:08.116357 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:08.110298    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" podStartSLOduration=2.33587471 podStartE2EDuration="3.110278054s" totalImagesPullingTime="774.403344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:17:08.106214865 +0000 UTC m=+2276.393433214" watchObservedRunningTime="2026-07-14 00:17:08.110278054 +0000 UTC m=+2276.397496401"
Jul 14 00:17:10.102850 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.102815    2558 generic.go:356] "Generic (PLEG): container finished" podID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerID="54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a" exitCode=0
Jul 14 00:17:10.103274 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.102906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerDied","Data":"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"}
Jul 14 00:17:10.103325 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.103283    2558 scope.go:117] "RemoveContainer" containerID="54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"
Jul 14 00:17:10.104264 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.104237    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:17:10.104318 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.104302    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:17:10.104361 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:10.104335    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 00:17:11.024631 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:11.024594    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:17:11.110005 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:11.109962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerStarted","Data":"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"}
Jul 14 00:17:11.680838 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:11.680809    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:17:11.682203 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:11.682175    2558 patch_prober.go:28] interesting pod/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 14 00:17:11.682319 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:11.682230    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 14 00:17:12.680652 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:12.680622    2558 patch_prober.go:28] interesting pod/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 14 00:17:12.681082 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:12.680993    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 14 00:17:13.680929 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:13.680876    2558 patch_prober.go:28] interesting pod/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 14 00:17:13.681370 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:13.680949    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 14 00:17:14.100947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:14.100920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:17:14.680266 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:14.680236    2558 patch_prober.go:28] interesting pod/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Jul 14 00:17:14.680448 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:14.680285    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Jul 14 00:17:15.684607 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:15.684575    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:17:15.684993 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:15.684850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:17:15.685590 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:17:15.685575    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:20:22.473945 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:20:22.473914    2558 scope.go:117] "RemoveContainer" containerID="7bd68dbab81e0d552a63b10c2758883979435c7102e48ee9100aba5dd6c89349"
Jul 14 00:21:22.493387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:21:22.493357    2558 scope.go:117] "RemoveContainer" containerID="f1d91eaed752f61a5bc1e4e3cbd22ce5ea7f734c2ce7cdc7f7bc5640bbcc6d70"
Jul 14 00:23:33.921751 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:33.921714    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"]
Jul 14 00:23:33.925143 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:33.925120    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:33.927452 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:33.927433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:23:33.933360 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:33.933337    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"]
Jul 14 00:23:34.040431 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:34.040400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8\" (UID: \"ba410bad-98b0-4484-97eb-0ddeb5795830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:34.141602 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:34.141572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8\" (UID: \"ba410bad-98b0-4484-97eb-0ddeb5795830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:34.142144 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:34.142127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8\" (UID: \"ba410bad-98b0-4484-97eb-0ddeb5795830\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:34.238297 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:34.238236    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:34.369402 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:34.369379    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"]
Jul 14 00:23:34.371077 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:23:34.371049    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba410bad_98b0_4484_97eb_0ddeb5795830.slice/crio-37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc WatchSource:0}: Error finding container 37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc: Status 404 returned error can't find the container with id 37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc
Jul 14 00:23:35.095112 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:35.095081    2558 generic.go:356] "Generic (PLEG): container finished" podID="ba410bad-98b0-4484-97eb-0ddeb5795830" containerID="4ec4d03b055f029b99e91232ffd70253c5c821e17de057b1fdff7a8a55fbdcc8" exitCode=0
Jul 14 00:23:35.095481 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:35.095127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8" event={"ID":"ba410bad-98b0-4484-97eb-0ddeb5795830","Type":"ContainerDied","Data":"4ec4d03b055f029b99e91232ffd70253c5c821e17de057b1fdff7a8a55fbdcc8"}
Jul 14 00:23:35.095481 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:35.095184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8" event={"ID":"ba410bad-98b0-4484-97eb-0ddeb5795830","Type":"ContainerStarted","Data":"37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc"}
Jul 14 00:23:36.232029 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:36.232007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:36.362870 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:36.362793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca\") pod \"ba410bad-98b0-4484-97eb-0ddeb5795830\" (UID: \"ba410bad-98b0-4484-97eb-0ddeb5795830\") "
Jul 14 00:23:36.363216 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:36.363193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba410bad-98b0-4484-97eb-0ddeb5795830" (UID: "ba410bad-98b0-4484-97eb-0ddeb5795830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:23:36.463554 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:36.463527    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba410bad-98b0-4484-97eb-0ddeb5795830-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:23:37.108174 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:37.108135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8" event={"ID":"ba410bad-98b0-4484-97eb-0ddeb5795830","Type":"ContainerDied","Data":"37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc"}
Jul 14 00:23:37.108353 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:37.108197    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a44d00bfd3aed997adaaf03fcac260d5bbf09a4573eeca8ae1f5895af7f8fc"
Jul 14 00:23:37.108353 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:37.108148    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8"
Jul 14 00:23:44.326534 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.326499    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:23:44.326971 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.326783    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" podUID="595318e8-457b-49fa-8bbb-53889c9ddf3a" containerName="maas-api" containerID="cri-o://69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78" gracePeriod=30
Jul 14 00:23:44.583515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.583454    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:23:44.735649 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.735618    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") pod \"595318e8-457b-49fa-8bbb-53889c9ddf3a\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") "
Jul 14 00:23:44.735649 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.735651    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca\") pod \"595318e8-457b-49fa-8bbb-53889c9ddf3a\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") "
Jul 14 00:23:44.735846 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.735689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9krlc\" (UniqueName: \"kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc\") pod \"595318e8-457b-49fa-8bbb-53889c9ddf3a\" (UID: \"595318e8-457b-49fa-8bbb-53889c9ddf3a\") "
Jul 14 00:23:44.736117 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.736085    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "595318e8-457b-49fa-8bbb-53889c9ddf3a" (UID: "595318e8-457b-49fa-8bbb-53889c9ddf3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:23:44.737589 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.737570    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc" (OuterVolumeSpecName: "kube-api-access-9krlc") pod "595318e8-457b-49fa-8bbb-53889c9ddf3a" (UID: "595318e8-457b-49fa-8bbb-53889c9ddf3a"). InnerVolumeSpecName "kube-api-access-9krlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:23:44.737758 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.737743    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "595318e8-457b-49fa-8bbb-53889c9ddf3a" (UID: "595318e8-457b-49fa-8bbb-53889c9ddf3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:23:44.836387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.836319    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/595318e8-457b-49fa-8bbb-53889c9ddf3a-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:23:44.836387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.836343    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/595318e8-457b-49fa-8bbb-53889c9ddf3a-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:23:44.836387 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:44.836354    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krlc\" (UniqueName: \"kubernetes.io/projected/595318e8-457b-49fa-8bbb-53889c9ddf3a-kube-api-access-9krlc\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:23:45.150383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.150289    2558 generic.go:356] "Generic (PLEG): container finished" podID="595318e8-457b-49fa-8bbb-53889c9ddf3a" containerID="69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78" exitCode=0
Jul 14 00:23:45.150383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.150342    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"
Jul 14 00:23:45.150606 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.150385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" event={"ID":"595318e8-457b-49fa-8bbb-53889c9ddf3a","Type":"ContainerDied","Data":"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"}
Jul 14 00:23:45.150606 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.150454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm" event={"ID":"595318e8-457b-49fa-8bbb-53889c9ddf3a","Type":"ContainerDied","Data":"02f8f75b68c9dad8a253af6dbe18b5ca8d504e7a70324cc702c04df2120f0505"}
Jul 14 00:23:45.150606 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.150490    2558 scope.go:117] "RemoveContainer" containerID="69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"
Jul 14 00:23:45.159788 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.159772    2558 scope.go:117] "RemoveContainer" containerID="69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"
Jul 14 00:23:45.160113 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:23:45.160083    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78\": container with ID starting with 69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78 not found: ID does not exist" containerID="69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"
Jul 14 00:23:45.160182 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.160122    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78"} err="failed to get container status \"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78\": rpc error: code = NotFound desc = could not find container \"69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78\": container with ID starting with 69cb875499110d70a4548c30e807b8360875880d7e39ab4ab5a4eca27308aa78 not found: ID does not exist"
Jul 14 00:23:45.171986 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.171963    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:23:45.175207 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.175186    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7e917f-d6b4b9b9d-q9dxm"]
Jul 14 00:23:45.432555 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:45.432479    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595318e8-457b-49fa-8bbb-53889c9ddf3a" path="/var/lib/kubelet/pods/595318e8-457b-49fa-8bbb-53889c9ddf3a/volumes"
Jul 14 00:23:59.751080 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:59.751045    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:23:59.751561 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:23:59.751254    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" containerID="cri-o://e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060" gracePeriod=30
Jul 14 00:24:00.120749 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.120710    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"]
Jul 14 00:24:00.121206 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121187    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba410bad-98b0-4484-97eb-0ddeb5795830" containerName="revoke-keys"
Jul 14 00:24:00.121305 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121209    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba410bad-98b0-4484-97eb-0ddeb5795830" containerName="revoke-keys"
Jul 14 00:24:00.121305 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121230    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595318e8-457b-49fa-8bbb-53889c9ddf3a" containerName="maas-api"
Jul 14 00:24:00.121305 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121238    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="595318e8-457b-49fa-8bbb-53889c9ddf3a" containerName="maas-api"
Jul 14 00:24:00.121463 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121347    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba410bad-98b0-4484-97eb-0ddeb5795830" containerName="revoke-keys"
Jul 14 00:24:00.121463 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.121363    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="595318e8-457b-49fa-8bbb-53889c9ddf3a" containerName="maas-api"
Jul 14 00:24:00.124379 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.124358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:00.126978 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.126951    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzxsj\""
Jul 14 00:24:00.131730 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.131705    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"]
Jul 14 00:24:00.173129 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.173087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09117f-f546z\" (UID: \"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:00.274114 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.274082    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09117f-f546z\" (UID: \"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:00.274769 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.274749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-09117f-f546z\" (UID: \"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:00.438592 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.438504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:00.579266 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.579239    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"]
Jul 14 00:24:00.581689 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:24:00.581663    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49f5a1a_8ec5_4ead_bff3_f30c9027f9bc.slice/crio-878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b WatchSource:0}: Error finding container 878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b: Status 404 returned error can't find the container with id 878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b
Jul 14 00:24:00.687353 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.687316    2558 patch_prober.go:28] interesting pod/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 00:24:00.687515 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:00.687379    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 00:24:01.232538 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:01.232451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z" event={"ID":"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc","Type":"ContainerStarted","Data":"de9d18b4b7a0e5933c11a8f1c7ae033cb9390ff859650b6764e0e58075ef7f50"}
Jul 14 00:24:01.232538 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:01.232504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z" event={"ID":"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc","Type":"ContainerStarted","Data":"878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b"}
Jul 14 00:24:01.250638 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:01.250590    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z" podStartSLOduration=1.250574246 podStartE2EDuration="1.250574246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:24:01.248002951 +0000 UTC m=+2689.535221299" watchObservedRunningTime="2026-07-14 00:24:01.250574246 +0000 UTC m=+2689.537792591"
Jul 14 00:24:02.238628 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:02.238592    2558 generic.go:356] "Generic (PLEG): container finished" podID="a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc" containerID="de9d18b4b7a0e5933c11a8f1c7ae033cb9390ff859650b6764e0e58075ef7f50" exitCode=0
Jul 14 00:24:02.239100 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:02.238678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z" event={"ID":"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc","Type":"ContainerDied","Data":"de9d18b4b7a0e5933c11a8f1c7ae033cb9390ff859650b6764e0e58075ef7f50"}
Jul 14 00:24:03.379202 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:03.379175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:03.403233 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:03.403200    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca\") pod \"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc\" (UID: \"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc\") "
Jul 14 00:24:03.403962 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:03.403934    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc" (UID: "a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:24:03.505078 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:03.504996    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc-maas-api-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:04.250248 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:04.250175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z"
Jul 14 00:24:04.250465 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:04.250173    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-09117f-f546z" event={"ID":"a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc","Type":"ContainerDied","Data":"878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b"}
Jul 14 00:24:04.250465 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:04.250293    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="878d5a40ae7723878315706b163f15549463cb1c851f658adf84f90ce4f61f3b"
Jul 14 00:24:05.007067 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.007035    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:24:05.121520 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121425    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121520 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121509    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121535    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121559    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121602    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9t5h\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121666    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121741    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:24:05.121856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs\") pod \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\" (UID: \"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f\") "
Jul 14 00:24:05.122241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.121913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:24:05.122241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.122086    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.122241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.122105    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istiod-ca-cert\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.122241 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.122174    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:24:05.122393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.122240    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:24:05.122794 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.122763    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:24:05.124107 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.124070    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:24:05.124221 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.124089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 00:24:05.124454 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.124429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 00:24:05.124566 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.124445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h" (OuterVolumeSpecName: "kube-api-access-p9t5h") pod "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" (UID: "34bc8f41-d941-4d84-bf0e-4e8d0487cc3f"). InnerVolumeSpecName "kube-api-access-p9t5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:24:05.222781 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222746    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-envoy\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.222781 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222777    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-credential-socket\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.222781 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222787    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-workload-certs\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.223051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222795    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-token\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.223051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222803    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-podinfo\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.223051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222825    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-istio-data\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.223051 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.222835    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t5h\" (UniqueName: \"kubernetes.io/projected/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f-kube-api-access-p9t5h\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:05.257830 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.257796    2558 generic.go:356] "Generic (PLEG): container finished" podID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerID="e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060" exitCode=0
Jul 14 00:24:05.258012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.257849    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"
Jul 14 00:24:05.258012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.257880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerDied","Data":"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"}
Jul 14 00:24:05.258012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.257958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b" event={"ID":"34bc8f41-d941-4d84-bf0e-4e8d0487cc3f","Type":"ContainerDied","Data":"59941c3f8f3596b606847be3c25cba7fb0889af015d9ae51bd56886821a1d51f"}
Jul 14 00:24:05.258012 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.257980    2558 scope.go:117] "RemoveContainer" containerID="e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"
Jul 14 00:24:05.267084 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.267059    2558 scope.go:117] "RemoveContainer" containerID="54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"
Jul 14 00:24:05.274926 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.274909    2558 scope.go:117] "RemoveContainer" containerID="e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"
Jul 14 00:24:05.275174 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:24:05.275157    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060\": container with ID starting with e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060 not found: ID does not exist" containerID="e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"
Jul 14 00:24:05.275225 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.275181    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060"} err="failed to get container status \"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060\": rpc error: code = NotFound desc = could not find container \"e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060\": container with ID starting with e497a7ed0c476955bd4b9a514959db40f186e19f655f7b7dfc402f8b8c99c060 not found: ID does not exist"
Jul 14 00:24:05.275225 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.275199    2558 scope.go:117] "RemoveContainer" containerID="54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"
Jul 14 00:24:05.275425 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:24:05.275407    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a\": container with ID starting with 54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a not found: ID does not exist" containerID="54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"
Jul 14 00:24:05.275462 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.275433    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a"} err="failed to get container status \"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a\": rpc error: code = NotFound desc = could not find container \"54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a\": container with ID starting with 54ae232b1d33cb03f0e46eb679789c531202d0bde13f6c3bbf458e6f5897875a not found: ID does not exist"
Jul 14 00:24:05.279450 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.279430    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:24:05.282865 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.282847    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7e917f-openshift-default-76699fc9f5-scr9b"]
Jul 14 00:24:05.431833 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:05.431753    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" path="/var/lib/kubelet/pods/34bc8f41-d941-4d84-bf0e-4e8d0487cc3f/volumes"
Jul 14 00:24:10.520053 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.520012    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:24:10.520556 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.520277    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" podUID="c1f9ed61-63d0-4b17-93c5-2891166be705" containerName="maas-api" containerID="cri-o://e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972" gracePeriod=30
Jul 14 00:24:10.775191 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.775135    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:24:10.872987 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.872949    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca\") pod \"c1f9ed61-63d0-4b17-93c5-2891166be705\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") "
Jul 14 00:24:10.872987 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.872993    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xd4n\" (UniqueName: \"kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n\") pod \"c1f9ed61-63d0-4b17-93c5-2891166be705\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") "
Jul 14 00:24:10.873199 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.873126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") pod \"c1f9ed61-63d0-4b17-93c5-2891166be705\" (UID: \"c1f9ed61-63d0-4b17-93c5-2891166be705\") "
Jul 14 00:24:10.873397 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.873369    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1f9ed61-63d0-4b17-93c5-2891166be705" (UID: "c1f9ed61-63d0-4b17-93c5-2891166be705"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 00:24:10.873513 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.873454    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f9ed61-63d0-4b17-93c5-2891166be705-openshift-service-ca\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:10.875092 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.875064    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1f9ed61-63d0-4b17-93c5-2891166be705" (UID: "c1f9ed61-63d0-4b17-93c5-2891166be705"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 00:24:10.875172 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.875122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n" (OuterVolumeSpecName: "kube-api-access-5xd4n") pod "c1f9ed61-63d0-4b17-93c5-2891166be705" (UID: "c1f9ed61-63d0-4b17-93c5-2891166be705"). InnerVolumeSpecName "kube-api-access-5xd4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 00:24:10.974723 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.974675    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1f9ed61-63d0-4b17-93c5-2891166be705-maas-api-tls\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:10.974723 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:10.974717    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xd4n\" (UniqueName: \"kubernetes.io/projected/c1f9ed61-63d0-4b17-93c5-2891166be705-kube-api-access-5xd4n\") on node \"ip-10-0-128-174.ec2.internal\" DevicePath \"\""
Jul 14 00:24:11.289787 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.289755    2558 generic.go:356] "Generic (PLEG): container finished" podID="c1f9ed61-63d0-4b17-93c5-2891166be705" containerID="e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972" exitCode=0
Jul 14 00:24:11.290165 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.289799    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"
Jul 14 00:24:11.290165 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.289809    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" event={"ID":"c1f9ed61-63d0-4b17-93c5-2891166be705","Type":"ContainerDied","Data":"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"}
Jul 14 00:24:11.290165 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.289865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5" event={"ID":"c1f9ed61-63d0-4b17-93c5-2891166be705","Type":"ContainerDied","Data":"923f95ec2435e2d1eefc100d1d3e3c1171a8c50cb53fee7c8ced41a736531df3"}
Jul 14 00:24:11.290165 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.289899    2558 scope.go:117] "RemoveContainer" containerID="e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"
Jul 14 00:24:11.300747 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.300726    2558 scope.go:117] "RemoveContainer" containerID="e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"
Jul 14 00:24:11.301114 ip-10-0-128-174 kubenswrapper[2558]: E0714 00:24:11.301088    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972\": container with ID starting with e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972 not found: ID does not exist" containerID="e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"
Jul 14 00:24:11.301190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.301126    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972"} err="failed to get container status \"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972\": rpc error: code = NotFound desc = could not find container \"e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972\": container with ID starting with e16f7b9822ee43cf738b98abb477a5e204c2c0838fceaed5ddfea7f3d4cb9972 not found: ID does not exist"
Jul 14 00:24:11.313230 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.313205    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:24:11.316680 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.316654    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-09117f-6c6bdc85c4-bl2c5"]
Jul 14 00:24:11.431737 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:11.431707    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1f9ed61-63d0-4b17-93c5-2891166be705" path="/var/lib/kubelet/pods/c1f9ed61-63d0-4b17-93c5-2891166be705/volumes"
Jul 14 00:24:32.051921 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:32.051876    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c958c8fd5-6hcz7_693d369c-d984-4b3a-8ba1-1f1241b382d1/manager/0.log"
Jul 14 00:24:32.295872 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:32.295841    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lq7qn_c40e8682-71f9-4799-83b1-b48b0d9de463/manager/0.log"
Jul 14 00:24:32.413428 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:32.413359    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bf4db56c-7xd8q_d501f675-1528-402a-aabf-31a843897b39/copy-manifests/0.log"
Jul 14 00:24:32.420367 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:32.420335    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bf4db56c-7xd8q_d501f675-1528-402a-aabf-31a843897b39/manager/0.log"
Jul 14 00:24:33.073423 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:33.073381    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6dbd59cc95-nnlv7_a85adf18-0744-49e1-8b3d-9972351cfb6f/maas-api/0.log"
Jul 14 00:24:33.188691 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:33.188667    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733120-42kmf_c268ed84-8ec4-4aaf-8024-51472f5ff92e/cleanup/0.log"
Jul 14 00:24:33.325396 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:33.325318    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733135-blz7g_9b984051-4368-4bdc-9d79-6483bbe12618/cleanup/0.log"
Jul 14 00:24:33.825402 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:33.825377    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-1ad924-s8x9z_83ef65f3-bb2a-46fc-bf56-4ce1a38e2ec9/revoke-keys/0.log"
Jul 14 00:24:33.944930 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:33.944881    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-bb015d31-6992j_7ddc53f5-5906-408a-8b7b-88cb970adc70/revoke-keys/0.log"
Jul 14 00:24:34.081007 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.080936    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-36055a-xw7cn_904e4e3c-b8ae-42e8-a00a-fe4544e22834/revoke-keys/0.log"
Jul 14 00:24:34.211344 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.211316    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-00e043-9qz49_7ee8ef46-e860-4760-b037-23151e9b0e89/revoke-keys/0.log"
Jul 14 00:24:34.337411 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.337332    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-982b7aec-znfgg_47f0d55f-4e6f-4c1a-8e7f-dc854aea78f6/revoke-keys/0.log"
Jul 14 00:24:34.577836 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.577811    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ca846028-zxhwh_eb74759e-b3ac-40db-85ad-63d318248c3e/revoke-keys/0.log"
Jul 14 00:24:34.690263 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.690193    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-7e917f-z2js8_ba410bad-98b0-4484-97eb-0ddeb5795830/revoke-keys/0.log"
Jul 14 00:24:34.799284 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.799256    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-09117f-f546z_a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc/revoke-keys/0.log"
Jul 14 00:24:34.915976 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:34.915949    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4nkr5_9aa87a00-f087-4f38-8b36-bcdbf678d293/postgres/0.log"
Jul 14 00:24:36.357008 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:36.356951    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pgkl9_9c0db5e9-d628-4ebb-a184-f9083b4fce4b/manager/0.log"
Jul 14 00:24:36.468065 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:36.468028    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fmc6g_fd6f8026-bd7c-426b-b21a-3cfc5bf1f289/manager/0.log"
Jul 14 00:24:36.574192 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:36.574169    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qlmps_24a72699-6483-49f1-a93f-1ed954e2d3a0/kuadrant-console-plugin/0.log"
Jul 14 00:24:36.699333 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:36.699232    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hbhwf_76e3bf6e-608a-48f6-8768-e1f8d073e24d/registry-server/0.log"
Jul 14 00:24:36.826231 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:36.826190    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8frvx_73ca59af-1127-4868-9fca-4096e86e2f4f/manager/0.log"
Jul 14 00:24:37.060093 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:37.060054    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8b84l_d213ae4e-650d-4ad4-9a92-5822ddabdf84/manager/0.log"
Jul 14 00:24:37.536557 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:37.536530    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-k5p7r_6b4d1aec-49f0-457c-8cce-17dc69d2349c/discovery/0.log"
Jul 14 00:24:37.891279 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:37.891207    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/1.log"
Jul 14 00:24:38.023994 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:38.023963    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-8b64v_5aa48b93-a93d-4ac4-837d-4e980f7232af/payload-pre-processing/0.log"
Jul 14 00:24:38.154376 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:38.154298    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-5cj5s_f7121938-30a7-4af5-b35c-14a9e8ee0723/payload-processing/0.log"
Jul 14 00:24:38.624391 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:38.624366    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nlnnz_a81f08ec-9427-44d9-b226-fd5df0291af0/main/0.log"
Jul 14 00:24:38.853314 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:38.853280    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-z6mbh_db4872f5-1fa4-4c92-b144-065d2ab3fdaa/main/0.log"
Jul 14 00:24:46.022479 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:46.022446    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9fw4w_caccffa7-8996-48b5-8eb1-1e83de888826/global-pull-secret-syncer/0.log"
Jul 14 00:24:46.142829 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:46.142794    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s7blq_27f93aec-e9a2-4864-a8f9-30631d8a7a4b/konnectivity-agent/0.log"
Jul 14 00:24:46.169988 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:46.169957    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-174.ec2.internal_650485c6eb65fd2e734a511b4a97894e/haproxy/0.log"
Jul 14 00:24:51.146620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.146583    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pgkl9_9c0db5e9-d628-4ebb-a184-f9083b4fce4b/manager/0.log"
Jul 14 00:24:51.172161 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.172135    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fmc6g_fd6f8026-bd7c-426b-b21a-3cfc5bf1f289/manager/0.log"
Jul 14 00:24:51.198823 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.198804    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qlmps_24a72699-6483-49f1-a93f-1ed954e2d3a0/kuadrant-console-plugin/0.log"
Jul 14 00:24:51.254685 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.254657    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hbhwf_76e3bf6e-608a-48f6-8768-e1f8d073e24d/registry-server/0.log"
Jul 14 00:24:51.397939 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.397842    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8frvx_73ca59af-1127-4868-9fca-4096e86e2f4f/manager/0.log"
Jul 14 00:24:51.537006 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:51.536969    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8b84l_d213ae4e-650d-4ad4-9a92-5822ddabdf84/manager/0.log"
Jul 14 00:24:53.567092 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:53.567058    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjw8c_9263031a-45fa-44af-8f76-33d260f4dc08/node-exporter/0.log"
Jul 14 00:24:53.603055 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:53.603024    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjw8c_9263031a-45fa-44af-8f76-33d260f4dc08/kube-rbac-proxy/0.log"
Jul 14 00:24:53.642777 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:53.642715    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjw8c_9263031a-45fa-44af-8f76-33d260f4dc08/init-textfile/0.log"
Jul 14 00:24:57.237825 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.237781    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"]
Jul 14 00:24:57.238290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238251    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy"
Jul 14 00:24:57.238290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238270    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy"
Jul 14 00:24:57.238290 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238288    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238297    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238331    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f9ed61-63d0-4b17-93c5-2891166be705" containerName="maas-api"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238340    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f9ed61-63d0-4b17-93c5-2891166be705" containerName="maas-api"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238351    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc" containerName="revoke-keys"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238359    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc" containerName="revoke-keys"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238445    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="34bc8f41-d941-4d84-bf0e-4e8d0487cc3f" containerName="istio-proxy"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238455    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f9ed61-63d0-4b17-93c5-2891166be705" containerName="maas-api"
Jul 14 00:24:57.238498 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.238463    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49f5a1a-8ec5-4ead-bff3-f30c9027f9bc" containerName="revoke-keys"
Jul 14 00:24:57.241784 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.241768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.244343 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.244320    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbgg9\"/\"kube-root-ca.crt\""
Jul 14 00:24:57.244451 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.244343    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kbgg9\"/\"openshift-service-ca.crt\""
Jul 14 00:24:57.244451 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.244352    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kbgg9\"/\"default-dockercfg-5fbwc\""
Jul 14 00:24:57.251273 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.251252    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"]
Jul 14 00:24:57.260805 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.260773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngcg\" (UniqueName: \"kubernetes.io/projected/d476d7c7-aedf-4730-b859-f02701fd4ebb-kube-api-access-hngcg\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.260944 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.260819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-podres\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.261021 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.260948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-proc\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.261021 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.260984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-sys\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.261021 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.261014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-lib-modules\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362411 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-sys\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362411 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-lib-modules\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngcg\" (UniqueName: \"kubernetes.io/projected/d476d7c7-aedf-4730-b859-f02701fd4ebb-kube-api-access-hngcg\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-podres\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-sys\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-proc\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-lib-modules\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362620 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362608    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-podres\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.362857 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.362659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d476d7c7-aedf-4730-b859-f02701fd4ebb-proc\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.370855 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.370832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngcg\" (UniqueName: \"kubernetes.io/projected/d476d7c7-aedf-4730-b859-f02701fd4ebb-kube-api-access-hngcg\") pod \"perf-node-gather-daemonset-cpwzf\" (UID: \"d476d7c7-aedf-4730-b859-f02701fd4ebb\") " pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.552147 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.552101    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:57.682357 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:57.682295    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"]
Jul 14 00:24:57.684670 ip-10-0-128-174 kubenswrapper[2558]: W0714 00:24:57.684639    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd476d7c7_aedf_4730_b859_f02701fd4ebb.slice/crio-7a98f20695776f0e7bc72b837b2a9605285b75675e3cd8f7c2ad8d2db41b8336 WatchSource:0}: Error finding container 7a98f20695776f0e7bc72b837b2a9605285b75675e3cd8f7c2ad8d2db41b8336: Status 404 returned error can't find the container with id 7a98f20695776f0e7bc72b837b2a9605285b75675e3cd8f7c2ad8d2db41b8336
Jul 14 00:24:58.522379 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.522340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf" event={"ID":"d476d7c7-aedf-4730-b859-f02701fd4ebb","Type":"ContainerStarted","Data":"fe144791315d81f09820f87929fbae548dab86ba4c141d80dde13fcb6e26c09e"}
Jul 14 00:24:58.522803 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.522404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf" event={"ID":"d476d7c7-aedf-4730-b859-f02701fd4ebb","Type":"ContainerStarted","Data":"7a98f20695776f0e7bc72b837b2a9605285b75675e3cd8f7c2ad8d2db41b8336"}
Jul 14 00:24:58.522803 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.522437    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:24:58.538433 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.538390    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf" podStartSLOduration=1.538376403 podStartE2EDuration="1.538376403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:24:58.537164559 +0000 UTC m=+2746.824382906" watchObservedRunningTime="2026-07-14 00:24:58.538376403 +0000 UTC m=+2746.825594748"
Jul 14 00:24:58.865950 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.865853    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sxkvh_3e73bef9-456c-4491-b77f-7b4e95f0b355/dns/0.log"
Jul 14 00:24:58.887603 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.887576    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sxkvh_3e73bef9-456c-4491-b77f-7b4e95f0b355/kube-rbac-proxy/0.log"
Jul 14 00:24:58.933856 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:58.933831    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mmj6_48fd7c53-c383-4b60-a62b-82a34fbcc221/dns-node-resolver/0.log"
Jul 14 00:24:59.481924 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:59.481875    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29733120-x7prj_9ee18ed3-6769-456a-9a49-cff272f60555/image-pruner/0.log"
Jul 14 00:24:59.526695 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:59.526668    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c89c8cd49-9fvx8_9ff7262e-2b87-4582-a6c8-c66b9f43974b/registry/0.log"
Jul 14 00:24:59.587190 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:24:59.587163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jzqgh_a434d4f3-48e1-41f7-b82e-5481913fe081/node-ca/0.log"
Jul 14 00:25:00.742217 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:00.742179    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-k5p7r_6b4d1aec-49f0-457c-8cce-17dc69d2349c/discovery/0.log"
Jul 14 00:25:00.821253 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:00.821202    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/0.log"
Jul 14 00:25:00.879163 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:00.879120    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/1.log"
Jul 14 00:25:00.917068 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:00.917029    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-8b64v_5aa48b93-a93d-4ac4-837d-4e980f7232af/payload-pre-processing/0.log"
Jul 14 00:25:00.954947 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:00.954917    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-5cj5s_f7121938-30a7-4af5-b35c-14a9e8ee0723/payload-processing/0.log"
Jul 14 00:25:01.720083 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:01.720050    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kpnjw_99ef6df1-688a-4170-ad90-4cdfdab2528c/serve-healthcheck-canary/0.log"
Jul 14 00:25:02.265082 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:02.265052    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ds97_77753122-f06d-44ff-a1d4-dc65e5524159/kube-rbac-proxy/0.log"
Jul 14 00:25:02.284763 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:02.284736    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ds97_77753122-f06d-44ff-a1d4-dc65e5524159/exporter/0.log"
Jul 14 00:25:02.304833 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:02.304811    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7ds97_77753122-f06d-44ff-a1d4-dc65e5524159/extractor/0.log"
Jul 14 00:25:04.537401 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:04.537377    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kbgg9/perf-node-gather-daemonset-cpwzf"
Jul 14 00:25:04.882391 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:04.882296    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5c958c8fd5-6hcz7_693d369c-d984-4b3a-8ba1-1f1241b382d1/manager/0.log"
Jul 14 00:25:05.004729 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:05.004701    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lq7qn_c40e8682-71f9-4799-83b1-b48b0d9de463/manager/0.log"
Jul 14 00:25:05.027153 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:05.027127    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bf4db56c-7xd8q_d501f675-1528-402a-aabf-31a843897b39/manager/0.log"
Jul 14 00:25:05.045715 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:05.045693    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74bf4db56c-7xd8q_d501f675-1528-402a-aabf-31a843897b39/copy-manifests/0.log"
Jul 14 00:25:08.186611 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:08.186576    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-k5p7r_6b4d1aec-49f0-457c-8cce-17dc69d2349c/discovery/0.log"
Jul 14 00:25:08.234935 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:08.234880    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/0.log"
Jul 14 00:25:08.292923 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:08.292870    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/1.log"
Jul 14 00:25:08.331647 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:08.331618    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-8b64v_5aa48b93-a93d-4ac4-837d-4e980f7232af/payload-pre-processing/0.log"
Jul 14 00:25:08.361181 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:08.361150    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-5cj5s_f7121938-30a7-4af5-b35c-14a9e8ee0723/payload-processing/0.log"
Jul 14 00:25:09.980554 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:09.980517    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-k5p7r_6b4d1aec-49f0-457c-8cce-17dc69d2349c/discovery/0.log"
Jul 14 00:25:10.028799 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:10.028772    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/0.log"
Jul 14 00:25:10.087550 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:10.087520    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jtwnl_3ba0ef99-657e-4d31-bbfc-f0de7d28af9a/istio-proxy/1.log"
Jul 14 00:25:10.117572 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:10.117544    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-8b64v_5aa48b93-a93d-4ac4-837d-4e980f7232af/payload-pre-processing/0.log"
Jul 14 00:25:10.149195 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:10.149168    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-5cj5s_f7121938-30a7-4af5-b35c-14a9e8ee0723/payload-processing/0.log"
Jul 14 00:25:11.899109 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:11.899079    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z5wkq_7eb9e210-8467-458b-b364-437157f57a02/migrator/0.log"
Jul 14 00:25:11.918393 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:11.918372    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z5wkq_7eb9e210-8467-458b-b364-437157f57a02/graceful-termination/0.log"
Jul 14 00:25:13.295789 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.295764    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-79hzk_125b0bad-f492-43c8-9846-41efc776bb2d/kube-multus/0.log"
Jul 14 00:25:13.508709 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.508683    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/kube-multus-additional-cni-plugins/0.log"
Jul 14 00:25:13.528839 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.528815    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/egress-router-binary-copy/0.log"
Jul 14 00:25:13.549834 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.549763    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/cni-plugins/0.log"
Jul 14 00:25:13.569946 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.569924    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/bond-cni-plugin/0.log"
Jul 14 00:25:13.589657 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.589634    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/routeoverride-cni/0.log"
Jul 14 00:25:13.609148 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.609123    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/whereabouts-cni-bincopy/0.log"
Jul 14 00:25:13.630283 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:13.630263    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zqjf_e6cad8c8-84e9-42d8-b267-57e1201c00d4/whereabouts-cni/0.log"
Jul 14 00:25:14.020622 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:14.020596    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pqr8z_26c1f071-28f4-4448-9c76-bddfc89bf893/network-metrics-daemon/0.log"
Jul 14 00:25:14.037383 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:14.037360    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pqr8z_26c1f071-28f4-4448-9c76-bddfc89bf893/kube-rbac-proxy/0.log"
Jul 14 00:25:14.916900 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:14.916863    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwxsn_a1d7010e-7e90-481b-84ea-91a9b2c59b0f/ovn-controller/0.log"
Jul 14 00:25:14.935454 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:14.935427    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwxsn_a1d7010e-7e90-481b-84ea-91a9b2c59b0f/ovn-acl-logging/0.log"
Jul 14 00:25:14.959271 ip-10-0-128-174 kubenswrapper[2558]: I0714 00:25:14.959249    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwxsn_a1d7010e-7e90-481b-84ea-91a9b2c59b0f/kube-rbac-proxy-node/0.log"