Jul 27 16:56:20.051673 ip-10-0-129-84 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 16:56:20.460553 ip-10-0-129-84 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:56:20.460553 ip-10-0-129-84 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 16:56:20.460553 ip-10-0-129-84 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:56:20.460553 ip-10-0-129-84 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 16:56:20.460553 ip-10-0-129-84 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:56:20.461697 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.461629    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 16:56:20.468091 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468076    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.468091 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468090    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468093    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468097    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468100    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468104    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468107    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468117    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468120    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468123    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468126    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468129    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468131    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468134    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468136    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468139    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468142    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468145    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468147    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468150    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468152    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.468156 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468155    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468157    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468160    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468163    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468166    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468168    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468171    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468173    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468176    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468179    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468181    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468184    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468186    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468189    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468192    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468194    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468197    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468199    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468202    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468210    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.468660 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468212    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468215    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468217    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468220    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468222    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468225    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468227    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468230    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468233    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468235    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468238    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468240    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468243    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468245    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468250    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468252    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468255    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468257    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468260    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.469169 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468263    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468265    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468268    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468270    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468272    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468275    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468277    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468279    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468282    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468285    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468287    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468289    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468292    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468295    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468297    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468299    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468302    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468304    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468307    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468310    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.469625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468312    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468315    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468318    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468320    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468323    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468325    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468328    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468331    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468335    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468340    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468343    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468346    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468349    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468353    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468357    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468361    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468364    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468368    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.470113 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.468370    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469104    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469111    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469114    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469118    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469122    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469126    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469129    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469132    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469135    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469138    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469141    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469144    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469147    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469150    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469152    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469155    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469157    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469160    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.470567 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469162    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469165    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469168    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469171    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469173    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469176    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469179    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469181    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469184    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469187    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469189    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469191    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469194    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469197    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469199    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469202    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469204    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469207    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469209    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469212    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469214    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.471051 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469217    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469220    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469223    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469227    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469231    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469233    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469236    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469239    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469241    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469244    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469246    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469249    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469251    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469254    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469258    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469260    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469265    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469268    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469271    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.471583 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469274    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469277    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469280    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469282    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469286    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469288    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469291    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469294    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469297    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469299    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469302    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469305    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469307    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469310    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469314    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469316    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469319    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469322    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469325    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469327    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.472053 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469330    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469333    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469335    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469338    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469340    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469343    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469345    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469348    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469350    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469353    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469355    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469358    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469360    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469363    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469366    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469368    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469371    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469373    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469376    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469378    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.472538 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.469380    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470382    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470389    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470392    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470395    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470397    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470400    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470406    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470409    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470411    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470414    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470417    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470419    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470422    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470425    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470428    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470430    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470436    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470439    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470441    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.473044 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470444    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470446    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470449    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470452    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470454    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470457    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470459    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470462    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470464    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470467    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470469    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470472    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470475    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470477    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470480    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470482    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470485    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470488    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470490    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470493    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.473519 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470496    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470499    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470501    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470504    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470506    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470509    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470512    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470514    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470517    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470521    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470523    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470526    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470529    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470531    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470534    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470536    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470538    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470541    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470544    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.474015 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470546    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470549    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470551    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470554    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470556    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470559    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470561    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470564    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470566    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470569    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470571    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470574    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470577    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470580    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470584    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470588    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470592    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470594    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470597    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.474480 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470600    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470603    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470607    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470611    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470614    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470617    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470619    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470622    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470625    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470627    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470629    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470632    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470635    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470637    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470640    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470642    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470645    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470647    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470650    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.474949 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470652    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470655    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.470659    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470720    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470726    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470736    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470740    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470744    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470747    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470752    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470758    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470762    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470764    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470768    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470771    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470775    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470778    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470780    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470784    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470786    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470789    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470794    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470797    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 16:56:20.475402 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470800    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470803    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470806    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470810    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470813    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470816    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470819    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470822    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470825    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470828    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470831    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470834    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470838    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470841    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470844    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470847    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470850    2564 flags.go:64] FLAG: --enable-server="true"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470853    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470857    2564 flags.go:64] FLAG: --event-burst="100"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470862    2564 flags.go:64] FLAG: --event-qps="50"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470865    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470868    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470870    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470885    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 16:56:20.475962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470889    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470898    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470902    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470905    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470908    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470911    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470914    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470917    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470920    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470923    2564 flags.go:64] FLAG: --feature-gates=""
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470927    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470930    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470933    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470936    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470939    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470942    2564 flags.go:64] FLAG: --help="false"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470945    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470948    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470951    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470954    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470957    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470961    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470964    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 16:56:20.476537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470966    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470969    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470972    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470975    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470980    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470983    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470986    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470989    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470992    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470995    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.470998    2564 flags.go:64] FLAG: --lock-file=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471001    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471003    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471006    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471012    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471015    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471018    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471021    2564 flags.go:64] FLAG: --logging-format="text"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471023    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471027    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471030    2564 flags.go:64] FLAG: --manifest-url=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471032    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471036    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471039    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471043    2564 flags.go:64] FLAG: --max-pods="110"
Jul 27 16:56:20.477159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471046    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471049    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471052    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471055    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471058    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471061    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471064    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471071    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471074    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471079    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471082    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471086    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471091    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471094    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471097    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471100    2564 flags.go:64] FLAG: --port="10250"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471103    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471106    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b4988e6affb08065"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471109    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471112    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471115    2564 flags.go:64] FLAG: --register-node="true"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471118    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471121    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471124    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 27 16:56:20.477757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471127    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471130    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471133    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471136    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471139    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471142    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471145    2564 flags.go:64] FLAG: --runonce="false"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471148    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471151    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471154    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471157    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471160    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471163    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471166    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471169    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471171    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471174    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471179    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471182    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471185    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471189    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471192    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471197    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471200    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471203    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 16:56:20.478686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471207    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471210    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471213    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471216    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471219    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471222    2564 flags.go:64] FLAG: --v="2"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471225    2564 flags.go:64] FLAG: --version="false"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471229    2564 flags.go:64] FLAG: --vmodule=""
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471233    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471236    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471341    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471344    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471347    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471350    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471353    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471356    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471358    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471361    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471364    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471367    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471369    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471372    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.479562 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471375    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471378    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471383    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471387    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471390    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471393    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471396    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471398    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471401    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471404    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471407    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471409    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471412    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471414    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471417    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471419    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471422    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471425    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471427    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471430    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.480120 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471432    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471435    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471437    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471440    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471442    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471445    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471447    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471450    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471452    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471455    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471458    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471460    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471462    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471465    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471469    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471472    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471474    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471478    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471481    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471483    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.480615 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471486    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471489    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471492    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471494    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471497    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471500    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471504    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471506    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471510    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471512    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471515    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471518    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471520    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471523    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471525    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471528    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471531    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471533    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471536    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.481155 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471538    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471541    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471543    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471546    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471548    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471551    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471553    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471557    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471559    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471562    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471566    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471568    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471571    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471573    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471577    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471581    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471584    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471587    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471589    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471592    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.481625 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471595    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471598    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471600    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471603    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471605    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471608    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471610    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.471613    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.471621    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.479640    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.479654    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.479677    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.479686    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479740    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.482227 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479744    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479748    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479751    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479754    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479757    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479760    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479763    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479766    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479769    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479771    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479774    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479777    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479780    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479782    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479785    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479788    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479791    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479793    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479796    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479799    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.482582 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479801    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479804    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479807    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479809    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479812    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479814    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479817    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479820    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479823    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479826    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479829    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479831    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479834    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479837    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479839    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479842    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479844    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479848    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479852    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.483080 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479855    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479858    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479861    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479864    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479866    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479869    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479873    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479888    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479891    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479894    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479897    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479900    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479902    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479905    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479908    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479910    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479913    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479915    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479918    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479921    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.483533 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479924    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479927    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479929    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479932    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479934    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479938    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479941    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479944    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479947    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479949    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479952    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479955    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479957    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479960    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479962    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479965    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479968    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479970    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479973    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.484141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479976    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479979    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479981    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479984    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479986    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479989    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479991    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479994    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479996    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.479999    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480003    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480006    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480009    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480012    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480015    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480018    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480020    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480023    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480026    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.484646 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480028    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.480033    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480140    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480145    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480148    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480152    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480155    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480158    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480161    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480164    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480167    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480170    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480172    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480175    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480178    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:56:20.485116 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480180    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480183    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480185    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480188    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480190    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480193    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480196    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480198    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480201    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480204    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480206    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480209    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480211    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480214    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480217    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480219    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480222    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480224    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480227    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:56:20.485487 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480230    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480232    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480235    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480237    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480240    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480242    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480245    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480247    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480250    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480252    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480255    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480257    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480260    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480263    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480267    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480270    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480273    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480276    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480278    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480281    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:56:20.485975 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480283    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480286    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480288    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480291    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480294    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480296    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480299    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480301    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480304    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480306    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480309    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480311    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480314    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480316    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480319    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480322    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480325    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480327    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480329    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480332    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:56:20.486453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480334    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480337    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480339    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480342    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480344    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480347    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480350    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480353    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480356    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480360    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480362    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480366    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480369    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480371    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480374    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480377    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480380    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480384    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480386    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:56:20.486954 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480389    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480391    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480394    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480396    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480399    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480401    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480404    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:20.480406    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.480411    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.480499    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.483272    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.484656    2564 server.go:1019] "Starting client certificate rotation"
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.484751    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:56:20.487404 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.484787    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:56:20.487713 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.487086    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:56:20.490102 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.490089    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:56:20.498861 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.498834    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 27 16:56:20.501018 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.501004    2564 log.go:25] "Validated CRI v1 image API"
Jul 27 16:56:20.501801 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.501786    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 16:56:20.506222 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.506202    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 db135ef4-fa3a-44d4-b43c-8615bdcafb0f:/dev/nvme0n1p4 f0aaf2c5-b010-46fd-ac31-396f1799342b:/dev/nvme0n1p3]
Jul 27 16:56:20.506301 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.506219    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 16:56:20.511114 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.511018    2564 manager.go:216] Machine: {Timestamp:2026-07-27 16:56:20.50971499 +0000 UTC m=+0.342079969 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100189 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217927371d0d994f6b32f1a38c0043 SystemUUID:ec217927-371d-0d99-4f6b-32f1a38c0043 BootID:914e662e-f90d-44c2-bb1e-5d47e2397ec5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:96:89:06:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:96:89:06:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:45:ab:23:b2:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 16:56:20.511114 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.511104    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 16:56:20.511249 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.511167    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 16:56:20.512655 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.512627    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 16:56:20.512771 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.512655    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 16:56:20.513374 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.513364    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 16:56:20.513410 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.513376    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 16:56:20.513410 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.513383    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:56:20.513991 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.513982    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:56:20.515305 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.515293    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 16:56:20.517175 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517164    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:56:20.517283 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517273    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 16:56:20.517327 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517299    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 16:56:20.517327 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517316    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 16:56:20.517327 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517326    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 16:56:20.517448 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517334    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 27 16:56:20.517448 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517342    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 16:56:20.517723 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517707    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:56:20.517755 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.517741    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:56:20.518041 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.518023    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:56:20.521238 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.521217    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 16:56:20.523060 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.523045    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 16:56:20.523120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.523074    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 16:56:20.524610 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524599    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524614    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524619    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524625    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524632    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524638    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524645    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524650    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 16:56:20.524656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524657    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 16:56:20.524859 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524663    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 16:56:20.524859 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524671    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 16:56:20.524859 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.524676    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 16:56:20.525389 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.525380    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 16:56:20.525389 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.525388    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 16:56:20.528280 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.528267    2564 server.go:1268] "Started kubelet"
Jul 27 16:56:20.528411 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.528342    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 16:56:20.528471 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.528448    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 16:56:20.528519 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.528474    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 16:56:20.528976 ip-10-0-129-84 systemd[1]: Started Kubernetes Kubelet.
Jul 27 16:56:20.530110 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.530067    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 16:56:20.530292 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.530273    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 16:56:20.530366 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.530295    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 16:56:20.530366 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.530294    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 16:56:20.530758 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.530745    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 16:56:20.532974 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.532957    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 16:56:20.536487 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.536472    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 16:56:20.536558 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.536542    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 16:56:20.536621 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.536571    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:20.537088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537070    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 16:56:20.537088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537081    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 16:56:20.537247 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537094    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 16:56:20.537247 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537213    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 16:56:20.537247 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537222    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 16:56:20.537416 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537398    2564 factory.go:55] Registering systemd factory
Jul 27 16:56:20.537506 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537421    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 27 16:56:20.537626 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537614    2564 factory.go:153] Registering CRI-O factory
Jul 27 16:56:20.537745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537628    2564 factory.go:223] Registration of the crio container factory successfully
Jul 27 16:56:20.537745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537684    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 16:56:20.537745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537708    2564 factory.go:103] Registering Raw factory
Jul 27 16:56:20.537745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.537732    2564 manager.go:1196] Started watching for new ooms in manager
Jul 27 16:56:20.538622 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.538157    2564 manager.go:349] Starting recovery of all containers
Jul 27 16:56:20.539366 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.539338    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 16:56:20.540287 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.539183    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e98f945d95  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.528242069 +0000 UTC m=+0.360607035,LastTimestamp:2026-07-27 16:56:20.528242069 +0000 UTC m=+0.360607035,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.544174 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.544148    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 16:56:20.549346 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.549328    2564 manager.go:354] Recovery completed
Jul 27 16:56:20.550444 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.550419    2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 27 16:56:20.553201 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.553187    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.554902 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.554872    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.554960 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.554916    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.554960 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.554930    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.555405 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.555394    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 16:56:20.555405 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.555405    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 16:56:20.555491 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.555421    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:56:20.555491 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.555456    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.556035 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.556020    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.556094 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.556049    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.556094 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.556063    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.556908 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.556837    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e9912b2185  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.554899845 +0000 UTC m=+0.387264815,LastTimestamp:2026-07-27 16:56:20.554899845 +0000 UTC m=+0.387264815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.557037 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.557026    2564 policy_none.go:49] "None policy: Start"
Jul 27 16:56:20.557092 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.557043    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 16:56:20.557092 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.557053    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 16:56:20.558000 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.557990    2564 policy_none.go:47] "Start"
Jul 27 16:56:20.567756 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.567701    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e9912b7b23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.554922787 +0000 UTC m=+0.387287754,LastTimestamp:2026-07-27 16:56:20.554922787 +0000 UTC m=+0.387287754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.574620 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.574535    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e9912bab97  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.554935191 +0000 UTC m=+0.387300160,LastTimestamp:2026-07-27 16:56:20.554935191 +0000 UTC m=+0.387300160,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.582052 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.582036    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq24k"
Jul 27 16:56:20.587233 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.587143    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-84.ec2.internal.18c633e9912b2185\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e9912b2185  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.554899845 +0000 UTC m=+0.387264815,LastTimestamp:2026-07-27 16:56:20.556034176 +0000 UTC m=+0.388399147,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.591374 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.591357    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kq24k"
Jul 27 16:56:20.596722 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.596671    2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-84.ec2.internal.18c633e9912b7b23\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-84.ec2.internal.18c633e9912b7b23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-84.ec2.internal,UID:ip-10-0-129-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-84.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-84.ec2.internal,},FirstTimestamp:2026-07-27 16:56:20.554922787 +0000 UTC m=+0.387287754,LastTimestamp:2026-07-27 16:56:20.556056006 +0000 UTC m=+0.388420974,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-84.ec2.internal,}"
Jul 27 16:56:20.599451 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599438    2564 manager.go:337] "Starting Device Plugin manager"
Jul 27 16:56:20.599523 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.599470    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 16:56:20.599523 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599479    2564 server.go:87] "Starting device plugin registration server"
Jul 27 16:56:20.599687 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599674    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 16:56:20.599737 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599690    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 16:56:20.599794 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599775    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 16:56:20.599913 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599894    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 16:56:20.600011 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.599922    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 16:56:20.600196 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.600181    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 16:56:20.600258 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.600228    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:20.650946 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.650927    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 16:56:20.653026 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.652044    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 16:56:20.653026 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.652084    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 16:56:20.653026 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.652104    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 16:56:20.653026 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.652142    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 16:56:20.655256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.655237    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:20.700129 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.700110    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.700797 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.700782    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.700858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.700807    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.700858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.700819    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.700858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.700845    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.711854 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.711816    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.711854 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.711835    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-84.ec2.internal\": node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:20.745273 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.745256    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:20.752808 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.752788    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"]
Jul 27 16:56:20.752859 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.752845    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.753515 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.753502    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.753556 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.753531    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.753556 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.753543    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.754833 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.754822    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.754977 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.754965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.755014 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.754998    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.757801 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.757787    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.757858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.757814    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.757858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.757825    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.758378 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.758362    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.758445 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.758389    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.758445 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.758399    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.759537 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.759524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.759580 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.759547    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:56:20.760202 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.760182    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:56:20.760295 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.760210    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:56:20.760295 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.760224    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:56:20.788596 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.788578    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.793069 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.793053    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-84.ec2.internal\" not found" node="ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.838434 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.838409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.838496 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.838444    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.838496 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.838484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb729aa8976fe5911d6204f16973d9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"fb729aa8976fe5911d6204f16973d9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.845854 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.845837    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:20.939618 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.939666 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.939666 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb729aa8976fe5911d6204f16973d9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"fb729aa8976fe5911d6204f16973d9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.939731 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939691    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb729aa8976fe5911d6204f16973d9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-84.ec2.internal\" (UID: \"fb729aa8976fe5911d6204f16973d9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.939731 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939699    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.939731 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:20.939691    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70ba3209c375caaa72be4b59a9949bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal\" (UID: \"70ba3209c375caaa72be4b59a9949bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:20.946162 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:20.946139    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:21.046708 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.046656    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:21.094600 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.094433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:21.096790 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.096768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:21.147215 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.147192    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:21.247453 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.247430    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:21.348350 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.348300    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-84.ec2.internal\" not found"
Jul 27 16:56:21.371916 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.371891    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:21.436852 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.436825    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:21.453231 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.453210    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:56:21.455600 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.455585    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal"
Jul 27 16:56:21.466170 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.466153    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:56:21.484661 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.484635    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 16:56:21.484784 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.484762    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 16:56:21.484827 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.484806    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 16:56:21.518926 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.518900    2564 apiserver.go:52] "Watching apiserver"
Jul 27 16:56:21.536682 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.536663    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 16:56:21.538763 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.538745    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 16:56:21.539167 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.539144    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rs7xq","openshift-dns/node-resolver-s7mpd","openshift-image-registry/node-ca-4bhv6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal","openshift-multus/network-metrics-daemon-w54gf","openshift-network-operator/iptables-alerter-x8879","openshift-ovn-kubernetes/ovnkube-node-vjtlp","kube-system/global-pull-secret-syncer-8w67f","kube-system/konnectivity-agent-ljms5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j","openshift-multus/multus-additional-cni-plugins-flf6p","openshift-multus/multus-smp5d","openshift-network-diagnostics/network-check-target-tgn6f"]
Jul 27 16:56:21.540697 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.540682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.542331 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.542150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.545207 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.545182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.546521 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.546503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.546609 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.546563    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:21.546910 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.546887    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.547057 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.547043    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.547206 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.547191    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9vw9d\""
Jul 27 16:56:21.547301 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.547283    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.547683 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.547667    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.547981 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.547955    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 16:56:21.548578 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.548557    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.548578 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.548572    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5hxjn\""
Jul 27 16:56:21.548718 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.548580    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.548718 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.548617    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlz5d\""
Jul 27 16:56:21.549283 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.549264    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.549883 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.549863    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.550617 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.550601    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.550694 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.550669    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:21.551870 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.551851    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.552347 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.552318    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.552443 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.552357    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.552443 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.552370    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-55lfh\""
Jul 27 16:56:21.552443 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.552428    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 16:56:21.552612 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.552479    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.553213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.553174    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 16:56:21.553420 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.553400    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.553989 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.553970    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.553989 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.553987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 16:56:21.554131 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.554071    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 16:56:21.554131 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.554113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dsnbw\""
Jul 27 16:56:21.554227 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.554138    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 16:56:21.554884 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.554867    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.556288 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.556268    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.557505 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.557484    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:21.557571 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.557546    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:21.573464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573438    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 16:56:21.574040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bpn6\""
Jul 27 16:56:21.574040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573824    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 16:56:21.574040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573870    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.574040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573895    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 16:56:21.574040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.573930    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zqj6x\""
Jul 27 16:56:21.574336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574201    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.574336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574247    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 16:56:21.574336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574280    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 16:56:21.574549 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574534    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 16:56:21.574916 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 16:56:21.574991 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574916    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-br6tc\""
Jul 27 16:56:21.574991 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.574905    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2z82r\""
Jul 27 16:56:21.577330 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.577313    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 16:56:21.581990 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.581974    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 16:56:21.586737 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.586719    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:56:21.595674 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.595650    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:51:21 +0000 UTC" deadline="2028-02-14 11:34:12.031285489 +0000 UTC"
Jul 27 16:56:21.595674 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.595671    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13602h37m50.435617174s"
Jul 27 16:56:21.599908 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.599865    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-grpb5"
Jul 27 16:56:21.613667 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.613649    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-grpb5"
Jul 27 16:56:21.638081 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.638066    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 16:56:21.645378 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-socket-dir-parent\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.645474 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645390    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-bin\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.645474 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9lx\" (UniqueName: \"kubernetes.io/projected/35d96c01-af44-46fd-a9fb-d8ff6270d875-kube-api-access-hb9lx\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.645474 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cnibin\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.645474 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-os-release\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645475    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-hosts-file\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-var-lib-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9def2e5-9322-42c3-a447-d790db9c1471-ovn-node-metrics-cert\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-run\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645545    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-conf-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645561    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-iptables-alerter-script\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645575    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-etc-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645590    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-node-log\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.645658 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-log-socket\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-var-lib-kubelet\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-k8s-cni-cncf-io\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-etc-kubernetes\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-binary-copy\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-multus-daemon-config\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-kube-api-access-cd68q\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645915    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-multus\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-env-overrides\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.645964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-dbus\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-conf\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-tmp\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48c5ca51-b05b-45c3-bece-1db371d1e4f1-agent-certs\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-system-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b0005a1-5a7a-48df-b38a-cc69f15efd96-serviceca\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-kubelet\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-sys\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-device-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-tmp-dir\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-host-slash\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-systemd\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-cni-binary-copy\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646318    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-script-lib\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-kubelet-config\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-modprobe-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646405    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-etc-tuned\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-system-cni-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.646706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-os-release\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646548    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-kubelet\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646623    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-cnibin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trz7h\" (UniqueName: \"kubernetes.io/projected/051fb427-fb8b-4d69-a230-1932448bb041-kube-api-access-trz7h\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjq5q\" (UniqueName: \"kubernetes.io/projected/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-kube-api-access-qjq5q\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b0005a1-5a7a-48df-b38a-cc69f15efd96-host\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646746    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67j2k\" (UniqueName: \"kubernetes.io/projected/5b0005a1-5a7a-48df-b38a-cc69f15efd96-kube-api-access-67j2k\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-systemd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-config\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysconfig\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-kubernetes\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646877    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-netns\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hjt\" (UniqueName: \"kubernetes.io/projected/f9def2e5-9322-42c3-a447-d790db9c1471-kube-api-access-h9hjt\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647353 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.646976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-lib-modules\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647043    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-host\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25mk\" (UniqueName: \"kubernetes.io/projected/c481205c-7801-48c8-b8b2-8000a231522a-kube-api-access-n25mk\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647089    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lcj\" (UniqueName: \"kubernetes.io/projected/35809892-f704-4f06-9bd7-6af46b65cf3c-kube-api-access-f4lcj\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzbs\" (UniqueName: \"kubernetes.io/projected/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-kube-api-access-9bzbs\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647128    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-hostroot\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-netd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647160    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48c5ca51-b05b-45c3-bece-1db371d1e4f1-konnectivity-ca\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-bin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-systemd-units\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-slash\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-netns\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-ovn\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.647832 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-multus-certs\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.648371 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.648371 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.648371 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647429    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.648371 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.648371 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.647487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.724817 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.724792    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ba3209c375caaa72be4b59a9949bfa.slice/crio-86ad45f6bf4ee2285c05c2f2198b4810193b6203d1a48b9998dc69dde6caa1db WatchSource:0}: Error finding container 86ad45f6bf4ee2285c05c2f2198b4810193b6203d1a48b9998dc69dde6caa1db: Status 404 returned error can't find the container with id 86ad45f6bf4ee2285c05c2f2198b4810193b6203d1a48b9998dc69dde6caa1db
Jul 27 16:56:21.725325 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.725304    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb729aa8976fe5911d6204f16973d9d7.slice/crio-5e6e7e804006d1d77f79fafa39a1574437c3218c68ca766611eed279fe50be9c WatchSource:0}: Error finding container 5e6e7e804006d1d77f79fafa39a1574437c3218c68ca766611eed279fe50be9c: Status 404 returned error can't find the container with id 5e6e7e804006d1d77f79fafa39a1574437c3218c68ca766611eed279fe50be9c
Jul 27 16:56:21.730324 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.730308    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:56:21.748134 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.748241 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748144    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-binary-copy\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.748241 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748172    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-multus-daemon-config\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748241 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748200    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-kube-api-access-cd68q\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748263    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-multus\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-env-overrides\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-dbus\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-conf\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748338    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-tmp\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-multus\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48c5ca51-b05b-45c3-bece-1db371d1e4f1-agent-certs\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.748409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-conf\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-system-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-dbus\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-system-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b0005a1-5a7a-48df-b38a-cc69f15efd96-serviceca\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-kubelet\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748683    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-sys\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-device-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-sys\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-tmp-dir\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-host-slash\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-multus-daemon-config\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-binary-copy\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.748881 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748869    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-systemd\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-cni-binary-copy\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-script-lib\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-kubelet-config\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-host-slash\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-modprobe-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-etc-tuned\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-env-overrides\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-system-cni-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-tmp-dir\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-system-cni-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-os-release\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749215    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-kubelet\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-systemd\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.749679 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-kubelet\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-cnibin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trz7h\" (UniqueName: \"kubernetes.io/projected/051fb427-fb8b-4d69-a230-1932448bb041-kube-api-access-trz7h\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-modprobe-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjq5q\" (UniqueName: \"kubernetes.io/projected/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-kube-api-access-qjq5q\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b0005a1-5a7a-48df-b38a-cc69f15efd96-host\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6762120c-be3f-4667-adc9-a6a118e1ae83-kubelet-config\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67j2k\" (UniqueName: \"kubernetes.io/projected/5b0005a1-5a7a-48df-b38a-cc69f15efd96-kube-api-access-67j2k\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-systemd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-cnibin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749428    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-config\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749446    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/051fb427-fb8b-4d69-a230-1932448bb041-cni-binary-copy\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749455    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysconfig\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-device-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-kubernetes\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-os-release\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-kubernetes\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.750516 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-script-lib\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-netns\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hjt\" (UniqueName: \"kubernetes.io/projected/f9def2e5-9322-42c3-a447-d790db9c1471-kube-api-access-h9hjt\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.748957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-lib-modules\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-host\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-systemd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.749873    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.749935    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:22.249914466 +0000 UTC m=+2.082279446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b0005a1-5a7a-48df-b38a-cc69f15efd96-serviceca\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750022    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b0005a1-5a7a-48df-b38a-cc69f15efd96-host\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-lib-modules\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.749875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25mk\" (UniqueName: \"kubernetes.io/projected/c481205c-7801-48c8-b8b2-8000a231522a-kube-api-access-n25mk\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.751423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-host\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lcj\" (UniqueName: \"kubernetes.io/projected/35809892-f704-4f06-9bd7-6af46b65cf3c-kube-api-access-f4lcj\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750230    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-netns\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzbs\" (UniqueName: \"kubernetes.io/projected/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-kube-api-access-9bzbs\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-kubelet\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-hostroot\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-netd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48c5ca51-b05b-45c3-bece-1db371d1e4f1-konnectivity-ca\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-bin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750353    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9def2e5-9322-42c3-a447-d790db9c1471-ovnkube-config\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-systemd-units\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-hostroot\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-slash\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-netns\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750463    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysconfig\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-ovn\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-var-lib-cni-bin\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.752285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-netd\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-multus-certs\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750604    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-multus-certs\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.750680    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750703    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.750727    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:22.250711 +0000 UTC m=+2.083075964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750735    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-etc-sysctl-d\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-ovn-kubernetes\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-run-netns\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-socket-dir-parent\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-run-ovn\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753186 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-bin\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-systemd-units\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9lx\" (UniqueName: \"kubernetes.io/projected/35d96c01-af44-46fd-a9fb-d8ff6270d875-kube-api-access-hb9lx\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cnibin\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-cni-bin\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-host-slash\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-os-release\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-hosts-file\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cnibin\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.750998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-var-lib-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-os-release\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9def2e5-9322-42c3-a447-d790db9c1471-ovn-node-metrics-cert\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48c5ca51-b05b-45c3-bece-1db371d1e4f1-konnectivity-ca\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751080    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-run\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-socket-dir-parent\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-run\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-conf-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.753717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751167    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35809892-f704-4f06-9bd7-6af46b65cf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751168    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-iptables-alerter-script\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-conf-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-etc-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-hosts-file\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751245    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-node-log\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-etc-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-multus-cni-dir\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751277    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-log-socket\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751282    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-var-lib-openvswitch\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-var-lib-kubelet\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-node-log\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-k8s-cni-cncf-io\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9def2e5-9322-42c3-a447-d790db9c1471-log-socket\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-etc-kubernetes\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751403    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-etc-kubernetes\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/051fb427-fb8b-4d69-a230-1932448bb041-host-run-k8s-cni-cncf-io\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c481205c-7801-48c8-b8b2-8000a231522a-var-lib-kubelet\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.754296 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-iptables-alerter-script\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.754826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.751818    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.754826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.752332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-etc-tuned\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.754826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.752436    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c481205c-7801-48c8-b8b2-8000a231522a-tmp\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.754826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.753428    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9def2e5-9322-42c3-a447-d790db9c1471-ovn-node-metrics-cert\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.754826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.753680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48c5ca51-b05b-45c3-bece-1db371d1e4f1-agent-certs\") pod \"konnectivity-agent-ljms5\" (UID: \"48c5ca51-b05b-45c3-bece-1db371d1e4f1\") " pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.762483 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.762455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd68q\" (UniqueName: \"kubernetes.io/projected/1ca8f8ac-e2eb-4f5e-bdde-94799368f0de-kube-api-access-cd68q\") pod \"iptables-alerter-x8879\" (UID: \"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de\") " pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.764338 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.764318    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67j2k\" (UniqueName: \"kubernetes.io/projected/5b0005a1-5a7a-48df-b38a-cc69f15efd96-kube-api-access-67j2k\") pod \"node-ca-4bhv6\" (UID: \"5b0005a1-5a7a-48df-b38a-cc69f15efd96\") " pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.764895 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.764876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25mk\" (UniqueName: \"kubernetes.io/projected/c481205c-7801-48c8-b8b2-8000a231522a-kube-api-access-n25mk\") pod \"tuned-rs7xq\" (UID: \"c481205c-7801-48c8-b8b2-8000a231522a\") " pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.768123 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.768105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzbs\" (UniqueName: \"kubernetes.io/projected/2e4b5cd4-faef-42e9-b68e-aef4fd138b75-kube-api-access-9bzbs\") pod \"multus-additional-cni-plugins-flf6p\" (UID: \"2e4b5cd4-faef-42e9-b68e-aef4fd138b75\") " pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.769095 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.769078    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:21.769188 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.769104    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:21.769188 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.769117    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:21.769188 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:21.769179    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:22.269161716 +0000 UTC m=+2.101526695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:21.769559 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.769539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjq5q\" (UniqueName: \"kubernetes.io/projected/59367e1f-f4c4-4c3b-8180-0068fd1cd54f-kube-api-access-qjq5q\") pod \"node-resolver-s7mpd\" (UID: \"59367e1f-f4c4-4c3b-8180-0068fd1cd54f\") " pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.769847 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.769827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hjt\" (UniqueName: \"kubernetes.io/projected/f9def2e5-9322-42c3-a447-d790db9c1471-kube-api-access-h9hjt\") pod \"ovnkube-node-vjtlp\" (UID: \"f9def2e5-9322-42c3-a447-d790db9c1471\") " pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.770748 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.770725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9lx\" (UniqueName: \"kubernetes.io/projected/35d96c01-af44-46fd-a9fb-d8ff6270d875-kube-api-access-hb9lx\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:21.772028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.772009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trz7h\" (UniqueName: \"kubernetes.io/projected/051fb427-fb8b-4d69-a230-1932448bb041-kube-api-access-trz7h\") pod \"multus-smp5d\" (UID: \"051fb427-fb8b-4d69-a230-1932448bb041\") " pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.773535 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.773516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lcj\" (UniqueName: \"kubernetes.io/projected/35809892-f704-4f06-9bd7-6af46b65cf3c-kube-api-access-f4lcj\") pod \"aws-ebs-csi-driver-node-pr28j\" (UID: \"35809892-f704-4f06-9bd7-6af46b65cf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.843822 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.843800    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:21.862122 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.862072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rs7xq"
Jul 27 16:56:21.870216 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.870194    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc481205c_7801_48c8_b8b2_8000a231522a.slice/crio-08da3668a63e9a28aa65526ae777dec8c145aa5ebd34b8c9e7f199274e5aa515 WatchSource:0}: Error finding container 08da3668a63e9a28aa65526ae777dec8c145aa5ebd34b8c9e7f199274e5aa515: Status 404 returned error can't find the container with id 08da3668a63e9a28aa65526ae777dec8c145aa5ebd34b8c9e7f199274e5aa515
Jul 27 16:56:21.876239 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.876224    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7mpd"
Jul 27 16:56:21.884669 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.884639    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59367e1f_f4c4_4c3b_8180_0068fd1cd54f.slice/crio-2bd67be3b0e420dee0df5962704fed1a8a672db6ad1a3c1690e7229ce7b6c339 WatchSource:0}: Error finding container 2bd67be3b0e420dee0df5962704fed1a8a672db6ad1a3c1690e7229ce7b6c339: Status 404 returned error can't find the container with id 2bd67be3b0e420dee0df5962704fed1a8a672db6ad1a3c1690e7229ce7b6c339
Jul 27 16:56:21.891763 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.891748    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4bhv6"
Jul 27 16:56:21.898599 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.898581    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0005a1_5a7a_48df_b38a_cc69f15efd96.slice/crio-a43cd68cb7109e465c4956d5a80e081eae81f768f9f2a30abce8b571875d57e2 WatchSource:0}: Error finding container a43cd68cb7109e465c4956d5a80e081eae81f768f9f2a30abce8b571875d57e2: Status 404 returned error can't find the container with id a43cd68cb7109e465c4956d5a80e081eae81f768f9f2a30abce8b571875d57e2
Jul 27 16:56:21.913959 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.913931    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x8879"
Jul 27 16:56:21.922539 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.922520    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca8f8ac_e2eb_4f5e_bdde_94799368f0de.slice/crio-f112e6b9553ddcc8c207df7826a025f14a1e117e0d59ba61698b0d254d8605a4 WatchSource:0}: Error finding container f112e6b9553ddcc8c207df7826a025f14a1e117e0d59ba61698b0d254d8605a4: Status 404 returned error can't find the container with id f112e6b9553ddcc8c207df7826a025f14a1e117e0d59ba61698b0d254d8605a4
Jul 27 16:56:21.934549 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.934533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:21.938941 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.938922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:21.941855 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.941834    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9def2e5_9322_42c3_a447_d790db9c1471.slice/crio-99e7d40b1c1c89fb109a6b1e7058a921aa7a74f3dc572728251cf42963e12d26 WatchSource:0}: Error finding container 99e7d40b1c1c89fb109a6b1e7058a921aa7a74f3dc572728251cf42963e12d26: Status 404 returned error can't find the container with id 99e7d40b1c1c89fb109a6b1e7058a921aa7a74f3dc572728251cf42963e12d26
Jul 27 16:56:21.946009 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.945991    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c5ca51_b05b_45c3_bece_1db371d1e4f1.slice/crio-60af5e40a6ef2275b3635669c83d0a93b6ee5e94aa1038fac37360fed3880a9e WatchSource:0}: Error finding container 60af5e40a6ef2275b3635669c83d0a93b6ee5e94aa1038fac37360fed3880a9e: Status 404 returned error can't find the container with id 60af5e40a6ef2275b3635669c83d0a93b6ee5e94aa1038fac37360fed3880a9e
Jul 27 16:56:21.963971 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.963951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j"
Jul 27 16:56:21.964395 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.964377    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:21.972202 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.972180    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35809892_f704_4f06_9bd7_6af46b65cf3c.slice/crio-ba3285d120b4a4e2099e1cd86e39e0b8b7f174ffdce5a62d7377b8e508e85cc7 WatchSource:0}: Error finding container ba3285d120b4a4e2099e1cd86e39e0b8b7f174ffdce5a62d7377b8e508e85cc7: Status 404 returned error can't find the container with id ba3285d120b4a4e2099e1cd86e39e0b8b7f174ffdce5a62d7377b8e508e85cc7
Jul 27 16:56:21.977983 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.977966    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-flf6p"
Jul 27 16:56:21.983240 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:21.983225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smp5d"
Jul 27 16:56:21.985141 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.985123    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4b5cd4_faef_42e9_b68e_aef4fd138b75.slice/crio-0efb714cc0c964541fb9b048e37affba5e2f0aa29fdb3db064dc9337df3bee85 WatchSource:0}: Error finding container 0efb714cc0c964541fb9b048e37affba5e2f0aa29fdb3db064dc9337df3bee85: Status 404 returned error can't find the container with id 0efb714cc0c964541fb9b048e37affba5e2f0aa29fdb3db064dc9337df3bee85
Jul 27 16:56:21.990453 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:21.990434    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051fb427_fb8b_4d69_a230_1932448bb041.slice/crio-a86e34a9d19ae41271e855476c79867390f28763019a8cc2a32f3ed934bef22c WatchSource:0}: Error finding container a86e34a9d19ae41271e855476c79867390f28763019a8cc2a32f3ed934bef22c: Status 404 returned error can't find the container with id a86e34a9d19ae41271e855476c79867390f28763019a8cc2a32f3ed934bef22c
Jul 27 16:56:22.255460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.255417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:22.255698 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.255478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:22.255698 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.255589    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:22.255698 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.255601    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:22.255698 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.255656    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:23.255639677 +0000 UTC m=+3.088004652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:22.255698 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.255679    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:23.255668078 +0000 UTC m=+3.088033034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:22.355847 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.355813    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:22.356072 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.356051    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:22.356160 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.356077    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:22.356160 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.356086    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:22.356160 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:22.356144    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:23.356126267 +0000 UTC m=+3.188491244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:22.626508 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.615695    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:51:21 +0000 UTC" deadline="2028-04-23 08:14:32.237739762 +0000 UTC"
Jul 27 16:56:22.626508 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.615741    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15255h18m9.622004544s"
Jul 27 16:56:22.670685 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.669257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ljms5" event={"ID":"48c5ca51-b05b-45c3-bece-1db371d1e4f1","Type":"ContainerStarted","Data":"60af5e40a6ef2275b3635669c83d0a93b6ee5e94aa1038fac37360fed3880a9e"}
Jul 27 16:56:22.673478 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.672679    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bhv6" event={"ID":"5b0005a1-5a7a-48df-b38a-cc69f15efd96","Type":"ContainerStarted","Data":"a43cd68cb7109e465c4956d5a80e081eae81f768f9f2a30abce8b571875d57e2"}
Jul 27 16:56:22.675823 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.675762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rs7xq" event={"ID":"c481205c-7801-48c8-b8b2-8000a231522a","Type":"ContainerStarted","Data":"08da3668a63e9a28aa65526ae777dec8c145aa5ebd34b8c9e7f199274e5aa515"}
Jul 27 16:56:22.677140 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.677111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"70ba3209c375caaa72be4b59a9949bfa","Type":"ContainerStarted","Data":"86ad45f6bf4ee2285c05c2f2198b4810193b6203d1a48b9998dc69dde6caa1db"}
Jul 27 16:56:22.679362 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.679332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j" event={"ID":"35809892-f704-4f06-9bd7-6af46b65cf3c","Type":"ContainerStarted","Data":"ba3285d120b4a4e2099e1cd86e39e0b8b7f174ffdce5a62d7377b8e508e85cc7"}
Jul 27 16:56:22.683346 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.683321    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"99e7d40b1c1c89fb109a6b1e7058a921aa7a74f3dc572728251cf42963e12d26"}
Jul 27 16:56:22.685757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.685716    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x8879" event={"ID":"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de","Type":"ContainerStarted","Data":"f112e6b9553ddcc8c207df7826a025f14a1e117e0d59ba61698b0d254d8605a4"}
Jul 27 16:56:22.689423 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.689398    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7mpd" event={"ID":"59367e1f-f4c4-4c3b-8180-0068fd1cd54f","Type":"ContainerStarted","Data":"2bd67be3b0e420dee0df5962704fed1a8a672db6ad1a3c1690e7229ce7b6c339"}
Jul 27 16:56:22.694203 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.694180    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"fb729aa8976fe5911d6204f16973d9d7","Type":"ContainerStarted","Data":"5e6e7e804006d1d77f79fafa39a1574437c3218c68ca766611eed279fe50be9c"}
Jul 27 16:56:22.699869 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.699842    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smp5d" event={"ID":"051fb427-fb8b-4d69-a230-1932448bb041","Type":"ContainerStarted","Data":"a86e34a9d19ae41271e855476c79867390f28763019a8cc2a32f3ed934bef22c"}
Jul 27 16:56:22.707904 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.707879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerStarted","Data":"0efb714cc0c964541fb9b048e37affba5e2f0aa29fdb3db064dc9337df3bee85"}
Jul 27 16:56:22.984285 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:22.984253    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:23.268257 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.267920    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:23.268257 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.268129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:23.268257 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.268255    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:23.268529 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.268322    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:25.268301988 +0000 UTC m=+5.100666946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:23.268529 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.268386    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:23.268529 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.268420    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:25.268407794 +0000 UTC m=+5.100772753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:23.369543 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.369504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:23.369736 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.369717    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:23.369804 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.369744    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:23.369804 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.369758    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:23.369915 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.369819    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:25.369800261 +0000 UTC m=+5.202165234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:23.515220 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.515188    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:56:23.616783 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.616664    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:51:21 +0000 UTC" deadline="2027-12-27 11:27:57.639445398 +0000 UTC"
Jul 27 16:56:23.616783 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.616731    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12426h31m34.022719142s"
Jul 27 16:56:23.653094 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.653063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:23.653529 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.653207    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:23.653660 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.653640    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:23.653808 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.653752    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:23.653890 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:23.653823    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:23.653984 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:23.653901    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:25.283763 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.283723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:25.284325 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.283796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:25.284325 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.283899    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:25.284325 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.283940    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:25.284325 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.283970    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:29.283947562 +0000 UTC m=+9.116312541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:25.284325 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.284000    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:29.283983433 +0000 UTC m=+9.116348406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:25.385200 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.385166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:25.385387 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.385362    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:25.385387 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.385384    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:25.385511 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.385396    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:25.385511 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.385497    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:29.385470876 +0000 UTC m=+9.217835834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:25.653028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.652999    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:25.653209 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.653125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:25.653209 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.653145    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:25.653354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.653308    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:25.653354 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:25.653331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:25.653570 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:25.653419    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:27.652821 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:27.652787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:27.653300 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:27.652948    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:27.653300 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:27.653105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:27.653300 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:27.653215    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:27.653300 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:27.653299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:27.653506 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:27.653380    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:29.320757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.320536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:29.320757 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.320609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:29.320757 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.320737    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:29.321301 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.320799    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:37.320781194 +0000 UTC m=+17.153146164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:29.321301 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.320737    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:29.321301 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.321178    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:37.321161625 +0000 UTC m=+17.153526593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:29.421125 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.421096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:29.421286 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.421228    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:29.421286 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.421244    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:29.421286 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.421252    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:29.421439 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.421296    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:37.421283441 +0000 UTC m=+17.253648395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.653059    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.653186    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.653562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.653670    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:29.653720    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:29.653829 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:29.653790    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:31.652656 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:31.652615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:31.653106 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:31.652667    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:31.653106 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:31.652629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:31.653106 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:31.652750    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:31.653106 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:31.652836    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:31.653106 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:31.652936    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:33.653140 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:33.653069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:33.653140 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:33.653094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:33.653140 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:33.653094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:33.653668 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:33.653195    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:33.653668 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:33.653269    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:33.653668 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:33.653342    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:35.652756 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:35.652722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:35.652756 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:35.652742    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:35.653244 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:35.652735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:35.653244 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:35.652856    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:35.653244 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:35.652974    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:35.653244 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:35.653051    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:37.386669 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.386585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:37.386669 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.386655    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:37.387141 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.386742    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:37.387141 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.386778    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:37.387141 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.386808    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.38679397 +0000 UTC m=+33.219158928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:37.387141 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.386835    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.386817214 +0000 UTC m=+33.219182179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:37.487558 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.487526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:37.487730 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.487692    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:37.487730 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.487711    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:37.487730 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.487722    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:37.487872 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.487773    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:56:53.487757052 +0000 UTC m=+33.320122019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:37.652996 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.652953    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:37.653185 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.652994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:37.653185 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:37.653080    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:37.653185 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.653096    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:37.653366 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.653217    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:37.653366 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:37.653308    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:39.652925 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:39.652894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:39.653365 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:39.652946    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:39.653365 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:39.652997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:39.653365 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:39.653013    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:39.653365 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:39.653101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:39.653365 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:39.653172    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:41.653128 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.653103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:41.653372 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.653197    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:41.653372 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:41.653204    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:41.653372 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.653222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:41.653372 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:41.653286    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:41.653372 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:41.653358    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:41.752952 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.752917    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" event={"ID":"fb729aa8976fe5911d6204f16973d9d7","Type":"ContainerStarted","Data":"9563aa3cdc1879e8e56451a153cdcf4a7d251188d57f2bedc6e3c3f1d4031e94"}
Jul 27 16:56:41.754671 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.754645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smp5d" event={"ID":"051fb427-fb8b-4d69-a230-1932448bb041","Type":"ContainerStarted","Data":"6a520ef2b1ca2bd1916f07b365f520857beba5c9e7d0a41d2556edbb6fe44b4e"}
Jul 27 16:56:41.756157 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.756132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rs7xq" event={"ID":"c481205c-7801-48c8-b8b2-8000a231522a","Type":"ContainerStarted","Data":"4203e7fa9fbb5a392fd87357ae2ea40dbc7fbf9fe923d9a473fd237e2ceecc73"}
Jul 27 16:56:41.772706 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.772551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-84.ec2.internal" podStartSLOduration=20.77253249 podStartE2EDuration="20.77253249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:41.772095021 +0000 UTC m=+21.604459998" watchObservedRunningTime="2026-07-27 16:56:41.77253249 +0000 UTC m=+21.604897467"
Jul 27 16:56:41.814461 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.814214    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rs7xq" podStartSLOduration=1.299178999 podStartE2EDuration="20.814196583s" totalImagesPullingTime="19.515017584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:41.793138338 +0000 UTC m=+21.625503331" watchObservedRunningTime="2026-07-27 16:56:41.814196583 +0000 UTC m=+21.646561561"
Jul 27 16:56:41.814985 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:41.814949    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-smp5d" podStartSLOduration=1.404224103 podStartE2EDuration="20.81493603s" totalImagesPullingTime="19.410711927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:41.814293125 +0000 UTC m=+21.646658105" watchObservedRunningTime="2026-07-27 16:56:41.81493603 +0000 UTC m=+21.647301007"
Jul 27 16:56:42.759770 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.759597    2564 generic.go:356] "Generic (PLEG): container finished" podID="70ba3209c375caaa72be4b59a9949bfa" containerID="6b0c844097242d054f33b4b5204dba1b1c486db396555fdd75cd31d07a6aaf6d" exitCode=0
Jul 27 16:56:42.760542 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.759689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"70ba3209c375caaa72be4b59a9949bfa","Type":"ContainerDied","Data":"6b0c844097242d054f33b4b5204dba1b1c486db396555fdd75cd31d07a6aaf6d"}
Jul 27 16:56:42.761446 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.761421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j" event={"ID":"35809892-f704-4f06-9bd7-6af46b65cf3c","Type":"ContainerStarted","Data":"5e3ac6371ffbd8d21ea8e78f810ff5666f17a70bd618c9805edc6a31eceb7f9b"}
Jul 27 16:56:42.766250 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"7764acb5d5d7b83f6c13a79be9feab59f3b279d659f1dcd4d0172aba4effccf2"}
Jul 27 16:56:42.766343 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766276    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"c15c9695d5761b6f00ccd996947188bd0b2f7918c809bbd18ae951a5301c2437"}
Jul 27 16:56:42.766415 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766388    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"ba180e6af86d1fad3790828bc93d287cda151ab16f81f29fa3a252ffc15d754d"}
Jul 27 16:56:42.766534 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766454    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"4222fa0d469282d6977f1c2346c12fc1359eb696dc5aaaba9015231a95c6e86a"}
Jul 27 16:56:42.766534 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"9e832aeab35b82e4aed9b4358b5b6c372ee5d563aad693b4b050481b66b8bc90"}
Jul 27 16:56:42.766631 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.766550    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"947c4ec65f9c1c2eef512fb1c8eb144f99b69d3767a1ba6e5aa6eef207096d93"}
Jul 27 16:56:42.768517 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.768488    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x8879" event={"ID":"1ca8f8ac-e2eb-4f5e-bdde-94799368f0de","Type":"ContainerStarted","Data":"cc06e741dc25156499ac17ca0103218042f5792c45cb0a892e67f02528feed8e"}
Jul 27 16:56:42.769917 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.769894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7mpd" event={"ID":"59367e1f-f4c4-4c3b-8180-0068fd1cd54f","Type":"ContainerStarted","Data":"b80a9e83350fadf1320c44c1791caa7ce09d7d237810d5b9184a14514170c679"}
Jul 27 16:56:42.771482 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.771459    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="8b7213320a61c68e6e87bc6046b80f4ed3209226cb9d043458ed516393558a99" exitCode=0
Jul 27 16:56:42.771587 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.771545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"8b7213320a61c68e6e87bc6046b80f4ed3209226cb9d043458ed516393558a99"}
Jul 27 16:56:42.773005 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.772905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ljms5" event={"ID":"48c5ca51-b05b-45c3-bece-1db371d1e4f1","Type":"ContainerStarted","Data":"b0f3ea882fdef5e1b5b56422e7c53e93a39e73b6e9fb6e8bcdcc2f086663c4ae"}
Jul 27 16:56:42.774364 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.774342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4bhv6" event={"ID":"5b0005a1-5a7a-48df-b38a-cc69f15efd96","Type":"ContainerStarted","Data":"5169377fc0fef6ce90d8f10e5c7b0e2908c1cf33531037ee306baa921395d6b8"}
Jul 27 16:56:42.850708 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.850671    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s7mpd" podStartSLOduration=2.35015536 podStartE2EDuration="21.850655734s" totalImagesPullingTime="19.500500374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:42.832161947 +0000 UTC m=+22.664526933" watchObservedRunningTime="2026-07-27 16:56:42.850655734 +0000 UTC m=+22.683020710"
Jul 27 16:56:42.850810 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.850739    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4bhv6" podStartSLOduration=6.616907003 podStartE2EDuration="21.850733612s" totalImagesPullingTime="15.233826609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:42.850220795 +0000 UTC m=+22.682585790" watchObservedRunningTime="2026-07-27 16:56:42.850733612 +0000 UTC m=+22.683098587"
Jul 27 16:56:42.868370 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:42.868335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ljms5" podStartSLOduration=2.448915604 podStartE2EDuration="21.86832151s" totalImagesPullingTime="19.419405906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:42.868186044 +0000 UTC m=+22.700551020" watchObservedRunningTime="2026-07-27 16:56:42.86832151 +0000 UTC m=+22.700686477"
Jul 27 16:56:43.412940 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.412762    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 16:56:43.615330 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.615199    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T16:56:43.412922791Z","UUID":"fd825dfd-2c05-4a6d-a1cd-4d1aea06b1de","Handler":null,"Name":"","Endpoint":""}
Jul 27 16:56:43.616398 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.616381    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 16:56:43.616509 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.616418    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 16:56:43.652636 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.652617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:43.652733 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.652617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:43.652733 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:43.652710    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:43.652814 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.652621    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:43.652814 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:43.652777    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:43.652912 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:43.652847    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:43.777563 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.777536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j" event={"ID":"35809892-f704-4f06-9bd7-6af46b65cf3c","Type":"ContainerStarted","Data":"923ac9c0d7cf49c9be082eb09d253318bde555b4b5f25d539284b14b2beb8a5c"}
Jul 27 16:56:43.779088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.779064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" event={"ID":"70ba3209c375caaa72be4b59a9949bfa","Type":"ContainerStarted","Data":"b2ad21dd3547b470003dfe3176cb916a3f4a58a1e15882b238621a24a0d5ba18"}
Jul 27 16:56:43.796982 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.796948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x8879" podStartSLOduration=3.335797807 podStartE2EDuration="22.796934863s" totalImagesPullingTime="19.461137056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:42.88923347 +0000 UTC m=+22.721598446" watchObservedRunningTime="2026-07-27 16:56:43.796934863 +0000 UTC m=+23.629299839"
Jul 27 16:56:43.797636 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:43.797610    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-84.ec2.internal" podStartSLOduration=22.797599274 podStartE2EDuration="22.797599274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:43.797009912 +0000 UTC m=+23.629374887" watchObservedRunningTime="2026-07-27 16:56:43.797599274 +0000 UTC m=+23.629964247"
Jul 27 16:56:45.652372 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.652342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:45.652908 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.652390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:45.652908 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.652342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:45.652908 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:45.652469    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:45.652908 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:45.652557    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:45.652908 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:45.652627    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:45.784702 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.784670    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j" event={"ID":"35809892-f704-4f06-9bd7-6af46b65cf3c","Type":"ContainerStarted","Data":"35d4ddb29a49bc22150efb7322e610d4d5088bcc756c84b38ff025ec05f5023d"}
Jul 27 16:56:45.788662 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.788627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"ee15673a1955a36d424fe15b07d4298d34e7aa8db835377f1e870ff0bbfcf9d0"}
Jul 27 16:56:45.804340 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:45.804299    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr28j" podStartSLOduration=2.7292466810000002 podStartE2EDuration="24.804285494s" totalImagesPullingTime="22.075038813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:45.804256761 +0000 UTC m=+25.636621741" watchObservedRunningTime="2026-07-27 16:56:45.804285494 +0000 UTC m=+25.636650472"
Jul 27 16:56:46.390465 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:46.390437    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:46.391126 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:46.391103    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:47.653297 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.653103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:47.653870 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.653151    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:47.653870 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:47.653391    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:47.653870 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.653151    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:47.653870 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:47.653472    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:47.653870 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:47.653556    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:47.798012 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.797976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" event={"ID":"f9def2e5-9322-42c3-a447-d790db9c1471","Type":"ContainerStarted","Data":"f631c9ccb0d729fd21692430007e5b9f10c762dd279cd3fcb96c0ea885767c4c"}
Jul 27 16:56:47.798388 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.798366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:47.814271 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.814249    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:47.836363 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:47.836325    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp" podStartSLOduration=6.821193456 podStartE2EDuration="26.836311455s" totalImagesPullingTime="20.015117999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:47.836217464 +0000 UTC m=+27.668582481" watchObservedRunningTime="2026-07-27 16:56:47.836311455 +0000 UTC m=+27.668676454"
Jul 27 16:56:48.461568 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.461530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:48.461715 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.461702    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 16:56:48.462212 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.462159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ljms5"
Jul 27 16:56:48.800477 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.800334    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:48.800477 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.800419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:48.817045 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:48.817018    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:56:49.094218 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.094143    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w54gf"]
Jul 27 16:56:49.094368 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.094256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:49.094368 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:49.094350    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:49.096582 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.096560    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w67f"]
Jul 27 16:56:49.096694 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.096640    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:49.096757 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:49.096705    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:49.099654 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.099634    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgn6f"]
Jul 27 16:56:49.099727 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.099712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:49.099792 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:49.099775    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:49.804201 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.804168    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="48fc3d700623cc4030e7ff39e2992ce0103a1b63b717b59bc64a16f2fbf42afa" exitCode=0
Jul 27 16:56:49.804858 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:49.804257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"48fc3d700623cc4030e7ff39e2992ce0103a1b63b717b59bc64a16f2fbf42afa"}
Jul 27 16:56:50.654551 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:50.654520    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:50.654695 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:50.654617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:50.654695 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:50.654659    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:50.654695 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:50.654649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:50.654839 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:50.654735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:50.654839 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:50.654820    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:52.652786 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:52.652637    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:52.653146 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:52.652637    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:52.653146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:52.652870    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn6f" podUID="417c753a-c6bc-455c-afd2-15fbe5411078"
Jul 27 16:56:52.653146 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:52.652671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:52.653146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:52.652974    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8w67f" podUID="6762120c-be3f-4667-adc9-a6a118e1ae83"
Jul 27 16:56:52.653146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:52.653027    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w54gf" podUID="35d96c01-af44-46fd-a9fb-d8ff6270d875"
Jul 27 16:56:52.811229 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:52.811200    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="5b31c870196c05986adab1a02c9142089c67b5d7a5db68eda456387d37becb9f" exitCode=0
Jul 27 16:56:52.811338 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:52.811260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"5b31c870196c05986adab1a02c9142089c67b5d7a5db68eda456387d37becb9f"}
Jul 27 16:56:53.417193 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:53.417162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:53.417326 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:53.417208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:53.417326 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.417291    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:53.417326 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.417292    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:53.417412 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.417336    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret podName:6762120c-be3f-4667-adc9-a6a118e1ae83 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:25.417323965 +0000 UTC m=+65.249688931 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret") pod "global-pull-secret-syncer-8w67f" (UID: "6762120c-be3f-4667-adc9-a6a118e1ae83") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:56:53.417412 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.417349    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:25.41734355 +0000 UTC m=+65.249708504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:56:53.517445 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:53.517417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:53.517621 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.517604    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:56:53.517663 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.517628    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:56:53.517663 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.517641    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ph4f6 for pod openshift-network-diagnostics/network-check-target-tgn6f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:53.517735 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:53.517701    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6 podName:417c753a-c6bc-455c-afd2-15fbe5411078 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:25.517681394 +0000 UTC m=+65.350046368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ph4f6" (UniqueName: "kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6") pod "network-check-target-tgn6f" (UID: "417c753a-c6bc-455c-afd2-15fbe5411078") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:56:54.520920 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.520861    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-84.ec2.internal" event="NodeReady"
Jul 27 16:56:54.521300 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.520968    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 16:56:54.558030 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.558003    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:56:54.579213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.579184    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:56:54.579401 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.579222    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-86hzs"]
Jul 27 16:56:54.579401 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.579297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.582597 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.582579    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 16:56:54.583080 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.583063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 16:56:54.583142 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.583063    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh2hl\""
Jul 27 16:56:54.586362 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.586340    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 16:56:54.591904 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.591868    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 16:56:54.603356 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.603337    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7srl7"]
Jul 27 16:56:54.603477 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.603457    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.606708 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.606688    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkv78\""
Jul 27 16:56:54.606843 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.606822    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 16:56:54.606957 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.606941    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 16:56:54.623360 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.623340    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86hzs"]
Jul 27 16:56:54.623435 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.623370    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7srl7"]
Jul 27 16:56:54.623472 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.623433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.626283 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.626267    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84blb\""
Jul 27 16:56:54.626368 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.626343    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 16:56:54.626495 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.626477    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 16:56:54.626601 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.626477    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 16:56:54.652690 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.652672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:56:54.652770 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.652700    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:56:54.652770 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.652766    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:56:54.656017 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.655987    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:56:54.656017 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.655991    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:56:54.656221 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.656056    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5cwf9\""
Jul 27 16:56:54.656221 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.656069    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:56:54.656221 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.655993    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7rsp\""
Jul 27 16:56:54.656221 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.656115    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:56:54.727103 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727190 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727190 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727190 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4adb49-ee35-4f55-b2b6-36b2510f5080-cert\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.727314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727218    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwxx\" (UniqueName: \"kubernetes.io/projected/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-kube-api-access-nxwxx\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.727314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqz9\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727291    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.727538 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727314    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727538 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.727538 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727383    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-config-volume\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.727538 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-tmp-dir\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.727538 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.727433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgw4\" (UniqueName: \"kubernetes.io/projected/bf4adb49-ee35-4f55-b2b6-36b2510f5080-kube-api-access-kjgw4\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.760456 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.760300    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:54.817930 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.817867    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="20c04dc13c504bd1d2877ff7c98de9980fe8d18eacc7f75864b34fbc76287edd" exitCode=0
Jul 27 16:56:54.817930 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.817914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"20c04dc13c504bd1d2877ff7c98de9980fe8d18eacc7f75864b34fbc76287edd"}
Jul 27 16:56:54.827911 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.827874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.827931    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.827958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.827978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828194 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:54.828058    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:56:54.828194 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-config-volume\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828194 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:54.828125    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:56:55.328095759 +0000 UTC m=+35.160460732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:56:54.828194 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-tmp-dir\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828207    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgw4\" (UniqueName: \"kubernetes.io/projected/bf4adb49-ee35-4f55-b2b6-36b2510f5080-kube-api-access-kjgw4\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828273    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828358    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4adb49-ee35-4f55-b2b6-36b2510f5080-cert\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwxx\" (UniqueName: \"kubernetes.io/projected/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-kube-api-access-nxwxx\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828457    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828460 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-tmp-dir\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828497    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqz9\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:54.828548    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:54.828563    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:54.828614    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:56:55.328599613 +0000 UTC m=+35.160964580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-config-volume\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.828840 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.828822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.829466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.829441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.829623 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.829594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.832089 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.832068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.832174 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.832067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.832174 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.832117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf4adb49-ee35-4f55-b2b6-36b2510f5080-cert\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.837369 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.837348    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgw4\" (UniqueName: \"kubernetes.io/projected/bf4adb49-ee35-4f55-b2b6-36b2510f5080-kube-api-access-kjgw4\") pod \"ingress-canary-7srl7\" (UID: \"bf4adb49-ee35-4f55-b2b6-36b2510f5080\") " pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:54.838450 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.838418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqz9\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.838740 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.838722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:54.840053 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.840032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwxx\" (UniqueName: \"kubernetes.io/projected/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-kube-api-access-nxwxx\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:54.864304 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.864279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:54.931540 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:54.931522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7srl7"
Jul 27 16:56:55.132041 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:55.132019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7srl7"]
Jul 27 16:56:55.133948 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:56:55.133923    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4adb49_ee35_4f55_b2b6_36b2510f5080.slice/crio-090105b6c85369174620e7154aaf48feddf96bc43ba00bb112002606b0ddfa23 WatchSource:0}: Error finding container 090105b6c85369174620e7154aaf48feddf96bc43ba00bb112002606b0ddfa23: Status 404 returned error can't find the container with id 090105b6c85369174620e7154aaf48feddf96bc43ba00bb112002606b0ddfa23
Jul 27 16:56:55.332636 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:55.332560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:55.332636 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:55.332610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:55.332848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:55.332707    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:56:55.332848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:55.332708    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:56:55.332848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:55.332776    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:56:56.332754729 +0000 UTC m=+36.165119697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:56:55.332848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:55.332716    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:56:55.332848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:55.332823    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:56:56.332812374 +0000 UTC m=+36.165177327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:56:55.821214 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:55.821181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7srl7" event={"ID":"bf4adb49-ee35-4f55-b2b6-36b2510f5080","Type":"ContainerStarted","Data":"090105b6c85369174620e7154aaf48feddf96bc43ba00bb112002606b0ddfa23"}
Jul 27 16:56:56.338472 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:56.338434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:56.338663 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:56.338513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:56.338663 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:56.338589    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:56:56.338663 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:56.338613    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:56:56.338663 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:56.338615    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:56:56.338837 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:56.338672    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:56:58.338656359 +0000 UTC m=+38.171021315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:56:56.338837 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:56.338686    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:56:58.338679972 +0000 UTC m=+38.171044926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:56:56.638256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:56.638173    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:57.906601 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:57.906529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:58.008310 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:58.008271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:56:58.354994 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:58.354799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:56:58.355117 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:58.355059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:56:58.355117 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:58.354956    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:56:58.355232 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:58.355175    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:57:02.355143979 +0000 UTC m=+42.187508937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:56:58.355232 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:58.355179    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:56:58.355232 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:58.355193    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:56:58.355341 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:56:58.355240    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:57:02.355224167 +0000 UTC m=+42.187589136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:56:58.829475 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:58.829442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7srl7" event={"ID":"bf4adb49-ee35-4f55-b2b6-36b2510f5080","Type":"ContainerStarted","Data":"36413cd653289fb58fe7f78f6ffb8ed8e087ba2a371187b3db06c3d2210ebb63"}
Jul 27 16:56:58.848965 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:58.848915    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7srl7" podStartSLOduration=1.768282989 podStartE2EDuration="4.848896766s" totalImagesPullingTime="3.080613777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:58.847902095 +0000 UTC m=+38.680267094" watchObservedRunningTime="2026-07-27 16:56:58.848896766 +0000 UTC m=+38.681261743"
Jul 27 16:56:59.884026 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:56:59.883990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:01.266129 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:01.266074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:01.434432 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:01.434401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:01.839069 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:01.839031    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="50009c87f88a94b190e385d739e2284be3a155476dcddd04d9125c47825c6b6c" exitCode=0
Jul 27 16:57:01.839237 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:01.839072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"50009c87f88a94b190e385d739e2284be3a155476dcddd04d9125c47825c6b6c"}
Jul 27 16:57:02.385058 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:02.385028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:02.385081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:02.385170    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:02.385185    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:02.385247    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:57:10.385231939 +0000 UTC m=+50.217596905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:02.385169    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:57:02.385354 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:02.385326    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:57:10.385310822 +0000 UTC m=+50.217675790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:57:02.844367 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:02.844334    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e4b5cd4-faef-42e9-b68e-aef4fd138b75" containerID="a18f9eb4064bd4910c4fea8c0bcee9956c37f1f8229895b160d02fc643c0551c" exitCode=0
Jul 27 16:57:02.844525 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:02.844390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerDied","Data":"a18f9eb4064bd4910c4fea8c0bcee9956c37f1f8229895b160d02fc643c0551c"}
Jul 27 16:57:03.849812 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:03.849781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-flf6p" event={"ID":"2e4b5cd4-faef-42e9-b68e-aef4fd138b75","Type":"ContainerStarted","Data":"db4d128ceb97f6f4ec038a4e7136098073c16a4edb583077eda666e3d1602c8c"}
Jul 27 16:57:03.877406 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:03.877360    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-flf6p" podStartSLOduration=7.209105731 podStartE2EDuration="42.877344919s" totalImagesPullingTime="35.668239188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:03.876735476 +0000 UTC m=+43.709100452" watchObservedRunningTime="2026-07-27 16:57:03.877344919 +0000 UTC m=+43.709709895"
Jul 27 16:57:04.000719 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:04.000690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:04.060174 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:04.060150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:05.950226 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:05.950191    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:06.052598 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:06.052571    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log" limit=1024
Jul 27 16:57:10.438097 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:10.438061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:10.438118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:10.438216    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:10.438228    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:10.438247    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:10.438282    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.438267955 +0000 UTC m=+66.270632930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:57:10.438591 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:10.438314    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.438294498 +0000 UTC m=+66.270659483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:57:15.314426 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.314396    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5w7vz"]
Jul 27 16:57:15.317861 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.317845    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"]
Jul 27 16:57:15.317998 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.317976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.320788 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.320765    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:57:15.320975 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.320765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 16:57:15.320975 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.320805    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 16:57:15.321135 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.321026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 16:57:15.321191 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.321146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.321237 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.321207    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r7shh\""
Jul 27 16:57:15.324043 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.324022    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:57:15.324484 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.324467    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zz87d\""
Jul 27 16:57:15.324570 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.324546    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 16:57:15.326358 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.326006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 16:57:15.327430 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.327415    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 16:57:15.330214 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.330192    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"]
Jul 27 16:57:15.330842 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.330822    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5w7vz"]
Jul 27 16:57:15.374203 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-config\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.374299 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.374299 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmm7\" (UniqueName: \"kubernetes.io/projected/30c41d72-0d82-4f03-a4bd-19dd758244ae-kube-api-access-zlmm7\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.374299 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c41d72-0d82-4f03-a4bd-19dd758244ae-serving-cert\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.374299 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjcv\" (UniqueName: \"kubernetes.io/projected/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-kube-api-access-bhjcv\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.374446 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.374320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-trusted-ca\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.407972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.407948    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"]
Jul 27 16:57:15.412168 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.412150    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.415765 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.415748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 16:57:15.416648 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.416631    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 16:57:15.417096 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.417081    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n4hvx\""
Jul 27 16:57:15.417156 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.417099    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 16:57:15.417156 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.417147    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 16:57:15.428707 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.428689    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"]
Jul 27 16:57:15.475194 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.475279 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475202    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c41d72-0d82-4f03-a4bd-19dd758244ae-serving-cert\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.475279 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjcv\" (UniqueName: \"kubernetes.io/projected/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-kube-api-access-bhjcv\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.475279 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-trusted-ca\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.475279 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475263    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ca26a8a-542c-40f6-9af6-d06625d47632-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.475437 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-config\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.475437 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.475437 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475425    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnsp\" (UniqueName: \"kubernetes.io/projected/6ca26a8a-542c-40f6-9af6-d06625d47632-kube-api-access-vhnsp\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.475590 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.475458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmm7\" (UniqueName: \"kubernetes.io/projected/30c41d72-0d82-4f03-a4bd-19dd758244ae-kube-api-access-zlmm7\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.475590 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.475490    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:15.475590 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.475590    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:15.975565382 +0000 UTC m=+55.807930355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:15.476040 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.476020    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-config\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.476132 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.476115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30c41d72-0d82-4f03-a4bd-19dd758244ae-trusted-ca\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.478463 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.478445    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c41d72-0d82-4f03-a4bd-19dd758244ae-serving-cert\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.485536 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.485516    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmm7\" (UniqueName: \"kubernetes.io/projected/30c41d72-0d82-4f03-a4bd-19dd758244ae-kube-api-access-zlmm7\") pod \"console-operator-584dcb6b54-5w7vz\" (UID: \"30c41d72-0d82-4f03-a4bd-19dd758244ae\") " pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.485780 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.485759    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjcv\" (UniqueName: \"kubernetes.io/projected/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-kube-api-access-bhjcv\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.576614 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.576552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ca26a8a-542c-40f6-9af6-d06625d47632-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.576717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.576657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnsp\" (UniqueName: \"kubernetes.io/projected/6ca26a8a-542c-40f6-9af6-d06625d47632-kube-api-access-vhnsp\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.576717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.576684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.576791 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.576772    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:15.576849 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.576839    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:16.076822549 +0000 UTC m=+55.909187522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:15.577180 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.577162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6ca26a8a-542c-40f6-9af6-d06625d47632-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.585868 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.585851    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnsp\" (UniqueName: \"kubernetes.io/projected/6ca26a8a-542c-40f6-9af6-d06625d47632-kube-api-access-vhnsp\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:15.628565 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.628546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:15.746999 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.746967    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5w7vz"]
Jul 27 16:57:15.750307 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:15.750280    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c41d72_0d82_4f03_a4bd_19dd758244ae.slice/crio-616948651a6d70af0d25e6fe65ceede78f97b6af01f82347e5bcb5293cd15a99 WatchSource:0}: Error finding container 616948651a6d70af0d25e6fe65ceede78f97b6af01f82347e5bcb5293cd15a99: Status 404 returned error can't find the container with id 616948651a6d70af0d25e6fe65ceede78f97b6af01f82347e5bcb5293cd15a99
Jul 27 16:57:15.881173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.881107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" event={"ID":"30c41d72-0d82-4f03-a4bd-19dd758244ae","Type":"ContainerStarted","Data":"616948651a6d70af0d25e6fe65ceede78f97b6af01f82347e5bcb5293cd15a99"}
Jul 27 16:57:15.979098 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:15.979074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:15.979242 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.979223    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:15.979325 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:15.979294    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:16.979278108 +0000 UTC m=+56.811643102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:16.080255 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:16.080230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:16.080345 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:16.080330    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:16.080396 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:16.080386    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:17.080373584 +0000 UTC m=+56.912738551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:16.988499 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:16.988460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:16.988954 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:16.988618    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:16.988954 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:16.988705    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:18.988685081 +0000 UTC m=+58.821050050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:17.089688 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:17.089652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:17.089838 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:17.089785    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:17.089925 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:17.089849    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:19.089832155 +0000 UTC m=+58.922197124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:18.889393 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:18.889360    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/0.log"
Jul 27 16:57:18.889788 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:18.889404    2564 generic.go:356] "Generic (PLEG): container finished" podID="30c41d72-0d82-4f03-a4bd-19dd758244ae" containerID="aa5ca7f80d791bde49ada9b92ffd6fdbe5eb24055bb3865e6139a1de3269d82b" exitCode=255
Jul 27 16:57:18.889788 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:18.889483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" event={"ID":"30c41d72-0d82-4f03-a4bd-19dd758244ae","Type":"ContainerDied","Data":"aa5ca7f80d791bde49ada9b92ffd6fdbe5eb24055bb3865e6139a1de3269d82b"}
Jul 27 16:57:18.889788 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:18.889639    2564 scope.go:117] "RemoveContainer" containerID="aa5ca7f80d791bde49ada9b92ffd6fdbe5eb24055bb3865e6139a1de3269d82b"
Jul 27 16:57:19.001036 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.001010    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:19.001169 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:19.001152    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:19.001242 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:19.001227    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.00120472 +0000 UTC m=+62.833569681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:19.101766 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.101743    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:19.101849 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:19.101818    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:19.101901 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:19.101864    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.101852035 +0000 UTC m=+62.934216994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:19.893536 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.893509    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/1.log"
Jul 27 16:57:19.893940 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.893920    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/0.log"
Jul 27 16:57:19.893994 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.893950    2564 generic.go:356] "Generic (PLEG): container finished" podID="30c41d72-0d82-4f03-a4bd-19dd758244ae" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b" exitCode=255
Jul 27 16:57:19.894029 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.893989    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" event={"ID":"30c41d72-0d82-4f03-a4bd-19dd758244ae","Type":"ContainerDied","Data":"89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"}
Jul 27 16:57:19.894066 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.894054    2564 scope.go:117] "RemoveContainer" containerID="aa5ca7f80d791bde49ada9b92ffd6fdbe5eb24055bb3865e6139a1de3269d82b"
Jul 27 16:57:19.894218 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:19.894201    2564 scope.go:117] "RemoveContainer" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"
Jul 27 16:57:19.894382 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:19.894359    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:20.823784 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:20.823760    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vjtlp"
Jul 27 16:57:20.896922 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:20.896902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/1.log"
Jul 27 16:57:20.897253 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:20.897146    2564 scope.go:117] "RemoveContainer" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"
Jul 27 16:57:20.897303 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:20.897260    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:22.607309 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.607276    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mkt7j"]
Jul 27 16:57:22.610267 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.610250    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.613336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.613308    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 16:57:22.613503 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.613485    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bzm7g\""
Jul 27 16:57:22.614644 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.614626    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 16:57:22.614644 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.614638    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 16:57:22.614779 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.614634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 16:57:22.626115 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.626097    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mkt7j"]
Jul 27 16:57:22.728355 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.728329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gjg\" (UniqueName: \"kubernetes.io/projected/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-api-access-b7gjg\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.728456 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.728379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.728524 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.728503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-crio-socket\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.728580 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.728533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-data-volume\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.728580 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.728566    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.828874 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-crio-socket\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.828968 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-data-volume\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.828968 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.828968 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gjg\" (UniqueName: \"kubernetes.io/projected/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-api-access-b7gjg\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.828968 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-crio-socket\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.829146 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.828982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.829146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:22.829039    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:57:22.829146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:22.829050    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:22.829146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:22.829102    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.329088328 +0000 UTC m=+63.161453282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : configmap "kube-rbac-proxy" not found
Jul 27 16:57:22.829146 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:22.829117    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:23.329109487 +0000 UTC m=+63.161474441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:22.829396 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.829146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-data-volume\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:22.837588 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:22.837563    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gjg\" (UniqueName: \"kubernetes.io/projected/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-api-access-b7gjg\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:23.030100 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:23.030061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:23.030269 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.030214    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:23.030306 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.030293    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:31.030277808 +0000 UTC m=+70.862642762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:23.130616 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:23.130594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:23.130745 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.130721    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:23.130783 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.130773    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:31.130760241 +0000 UTC m=+70.963125195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:23.331791 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:23.331724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:23.331962 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:23.331789    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:23.331962 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.331855    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:57:23.331962 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.331913    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:23.331962 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.331945    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:24.331930776 +0000 UTC m=+64.164295733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : configmap "kube-rbac-proxy" not found
Jul 27 16:57:23.332151 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:23.331992    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:24.331972513 +0000 UTC m=+64.164337485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:24.340939 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:24.340901    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:24.341333 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:24.340985    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:24.341333 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:24.341015    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:57:24.341333 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:24.341075    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.341060668 +0000 UTC m=+66.173425627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : configmap "kube-rbac-proxy" not found
Jul 27 16:57:24.341333 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:24.341082    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:24.341333 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:24.341123    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:26.341111985 +0000 UTC m=+66.173476966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:24.377387 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:24.377361    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7mpd_59367e1f-f4c4-4c3b-8180-0068fd1cd54f/dns-node-resolver/0.log"
Jul 27 16:57:25.178069 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.178046    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bhv6_5b0005a1-5a7a-48df-b38a-cc69f15efd96/node-ca/0.log"
Jul 27 16:57:25.450382 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.450297    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:57:25.450820 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.450450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:57:25.453299 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.453279    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:57:25.453407 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.453284    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:57:25.460777 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:25.460756    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 16:57:25.460833 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:25.460822    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs podName:35d96c01-af44-46fd-a9fb-d8ff6270d875 nodeName:}" failed. No retries permitted until 2026-07-27 16:58:29.460803271 +0000 UTC m=+129.293168235 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs") pod "network-metrics-daemon-w54gf" (UID: "35d96c01-af44-46fd-a9fb-d8ff6270d875") : secret "metrics-daemon-secret" not found
Jul 27 16:57:25.462769 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.462743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6762120c-be3f-4667-adc9-a6a118e1ae83-original-pull-secret\") pod \"global-pull-secret-syncer-8w67f\" (UID: \"6762120c-be3f-4667-adc9-a6a118e1ae83\") " pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:57:25.551379 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.551339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:57:25.554203 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.554184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:57:25.564086 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.564069    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:57:25.566331 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.566315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8w67f"
Jul 27 16:57:25.574745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.574727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4f6\" (UniqueName: \"kubernetes.io/projected/417c753a-c6bc-455c-afd2-15fbe5411078-kube-api-access-ph4f6\") pod \"network-check-target-tgn6f\" (UID: \"417c753a-c6bc-455c-afd2-15fbe5411078\") " pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:57:25.628975 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.628873    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:25.628975 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.628967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:25.629286 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.629274    2564 scope.go:117] "RemoveContainer" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"
Jul 27 16:57:25.629462 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:25.629443    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:25.683264 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.683236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8w67f"]
Jul 27 16:57:25.685990 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:25.685960    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6762120c_be3f_4667_adc9_a6a118e1ae83.slice/crio-eaf3d96c867e2d54ca7be03771a267a6e57925aab0ddf5cdbf60cb75a7ed46f2 WatchSource:0}: Error finding container eaf3d96c867e2d54ca7be03771a267a6e57925aab0ddf5cdbf60cb75a7ed46f2: Status 404 returned error can't find the container with id eaf3d96c867e2d54ca7be03771a267a6e57925aab0ddf5cdbf60cb75a7ed46f2
Jul 27 16:57:25.865789 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.865710    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5cwf9\""
Jul 27 16:57:25.872687 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.872665    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:57:25.910595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.910564    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w67f" event={"ID":"6762120c-be3f-4667-adc9-a6a118e1ae83","Type":"ContainerStarted","Data":"eaf3d96c867e2d54ca7be03771a267a6e57925aab0ddf5cdbf60cb75a7ed46f2"}
Jul 27 16:57:25.988705 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:25.988680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgn6f"]
Jul 27 16:57:25.991405 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:25.991371    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417c753a_c6bc_455c_afd2_15fbe5411078.slice/crio-0615824dbf9624c2d2253fbf5aaf416fe8e7c4f54cd15f7c2de4a1bbb63473bc WatchSource:0}: Error finding container 0615824dbf9624c2d2253fbf5aaf416fe8e7c4f54cd15f7c2de4a1bbb63473bc: Status 404 returned error can't find the container with id 0615824dbf9624c2d2253fbf5aaf416fe8e7c4f54cd15f7c2de4a1bbb63473bc
Jul 27 16:57:26.356606 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.356576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:26.356766 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.356642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:26.356766 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.356711    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:57:26.356766 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.356745    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:26.356871 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.356780    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:30.356762934 +0000 UTC m=+70.189127902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : configmap "kube-rbac-proxy" not found
Jul 27 16:57:26.356871 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.356795    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:30.356789488 +0000 UTC m=+70.189154442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:26.389067 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.389034    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7srl7_bf4adb49-ee35-4f55-b2b6-36b2510f5080/serve-healthcheck-canary/0.log"
Jul 27 16:57:26.457482 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.457458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.457492    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.457589    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.457598    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-579689bb78-x6msx: secret "image-registry-tls" not found
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.457631    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls podName:9243670c-4cae-4243-a2cb-d58bc418150b nodeName:}" failed. No retries permitted until 2026-07-27 16:57:58.457620043 +0000 UTC m=+98.289984996 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls") pod "image-registry-579689bb78-x6msx" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b") : secret "image-registry-tls" not found
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.457589    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:57:26.457817 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:26.457705    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls podName:3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f nodeName:}" failed. No retries permitted until 2026-07-27 16:57:58.457692484 +0000 UTC m=+98.290057457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls") pod "dns-default-86hzs" (UID: "3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f") : secret "dns-default-metrics-tls" not found
Jul 27 16:57:26.916174 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:26.916101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgn6f" event={"ID":"417c753a-c6bc-455c-afd2-15fbe5411078","Type":"ContainerStarted","Data":"0615824dbf9624c2d2253fbf5aaf416fe8e7c4f54cd15f7c2de4a1bbb63473bc"}
Jul 27 16:57:29.926132 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:29.926092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgn6f" event={"ID":"417c753a-c6bc-455c-afd2-15fbe5411078","Type":"ContainerStarted","Data":"376492fdf34f884ce3fef44a33e0437dffeab211142343110f4c6af4c645e287"}
Jul 27 16:57:29.926561 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:29.926191    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:57:29.950063 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:29.949993    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgn6f" podStartSLOduration=65.964064379 podStartE2EDuration="1m8.949973962s" totalImagesPullingTime="2.985909583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:29.949898915 +0000 UTC m=+69.782263902" watchObservedRunningTime="2026-07-27 16:57:29.949973962 +0000 UTC m=+69.782338939"
Jul 27 16:57:30.392415 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:30.392339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:30.392567 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:30.392437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:30.392567 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:30.392478    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:57:30.392567 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:30.392558    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:38.392537844 +0000 UTC m=+78.224902816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : configmap "kube-rbac-proxy" not found
Jul 27 16:57:30.392689 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:30.392572    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:30.392689 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:30.392617    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:38.392601913 +0000 UTC m=+78.224966871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:30.930448 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:30.930417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8w67f" event={"ID":"6762120c-be3f-4667-adc9-a6a118e1ae83","Type":"ContainerStarted","Data":"2f0ca65a92a91ed2860429ab5b95298986d2f27345c880fe28cb1261d91255e4"}
Jul 27 16:57:30.950272 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:30.950225    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8w67f" podStartSLOduration=64.793174297 podStartE2EDuration="1m9.950211852s" totalImagesPullingTime="5.157037555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:30.948635734 +0000 UTC m=+70.781000709" watchObservedRunningTime="2026-07-27 16:57:30.950211852 +0000 UTC m=+70.782576827"
Jul 27 16:57:31.098009 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:31.097934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:31.098130 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:31.098064    2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 16:57:31.098130 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:31.098122    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls podName:771d4faa-ca73-4478-ac31-1ee76f3ef5ef nodeName:}" failed. No retries permitted until 2026-07-27 16:57:47.098107778 +0000 UTC m=+86.930472735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls") pod "cluster-samples-operator-6748489656-mf6bs" (UID: "771d4faa-ca73-4478-ac31-1ee76f3ef5ef") : secret "samples-operator-tls" not found
Jul 27 16:57:31.199116 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:31.199083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:31.199239 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:31.199178    2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:31.199239 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:31.199223    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls podName:6ca26a8a-542c-40f6-9af6-d06625d47632 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:47.199211164 +0000 UTC m=+87.031576151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kt6d2" (UID: "6ca26a8a-542c-40f6-9af6-d06625d47632") : secret "cluster-monitoring-operator-tls" not found
Jul 27 16:57:38.455714 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:38.455679    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:38.456219 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:38.455752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:38.456219 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:38.455915    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:38.456219 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:38.455984    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls podName:c95f73ba-bcd4-46d1-a5d1-6cb8a4300188 nodeName:}" failed. No retries permitted until 2026-07-27 16:57:54.45596597 +0000 UTC m=+94.288330928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mkt7j" (UID: "c95f73ba-bcd4-46d1-a5d1-6cb8a4300188") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:57:38.456327 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:38.456260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:39.652672 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.652642    2564 scope.go:117] "RemoveContainer" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"
Jul 27 16:57:39.953358 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.953303    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 16:57:39.953777 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.953762    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/1.log"
Jul 27 16:57:39.953833 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.953795    2564 generic.go:356] "Generic (PLEG): container finished" podID="30c41d72-0d82-4f03-a4bd-19dd758244ae" containerID="f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe" exitCode=255
Jul 27 16:57:39.953868 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.953835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" event={"ID":"30c41d72-0d82-4f03-a4bd-19dd758244ae","Type":"ContainerDied","Data":"f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe"}
Jul 27 16:57:39.953919 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.953899    2564 scope.go:117] "RemoveContainer" containerID="89c3b68cd8caea97c23f72b20d523da0106dba104828b3e0c7da7efcc1d1124b"
Jul 27 16:57:39.954173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:39.954156    2564 scope.go:117] "RemoveContainer" containerID="f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe"
Jul 27 16:57:39.954338 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:39.954319    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:40.957549 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:40.957524    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 16:57:45.629409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:45.629306    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:45.629409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:45.629353    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:57:45.629805 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:45.629646    2564 scope.go:117] "RemoveContainer" containerID="f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe"
Jul 27 16:57:45.629848 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:45.629818    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:47.124403 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.124370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:47.126931 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.126903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/771d4faa-ca73-4478-ac31-1ee76f3ef5ef-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-mf6bs\" (UID: \"771d4faa-ca73-4478-ac31-1ee76f3ef5ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:47.138095 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.138072    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zz87d\""
Jul 27 16:57:47.144980 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.144965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"
Jul 27 16:57:47.225522 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.225493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:47.227870 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.227848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6ca26a8a-542c-40f6-9af6-d06625d47632-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kt6d2\" (UID: \"6ca26a8a-542c-40f6-9af6-d06625d47632\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:47.270739 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.270718    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs"]
Jul 27 16:57:47.524142 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.524096    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n4hvx\""
Jul 27 16:57:47.531754 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.531724    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"
Jul 27 16:57:47.654049 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.652540    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2"]
Jul 27 16:57:47.976180 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.976140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2" event={"ID":"6ca26a8a-542c-40f6-9af6-d06625d47632","Type":"ContainerStarted","Data":"0fcac297712aafef5c7825270011d89bdd4b9cd4a28ed9527fd6aa4b92d9363b"}
Jul 27 16:57:47.977160 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:47.977139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs" event={"ID":"771d4faa-ca73-4478-ac31-1ee76f3ef5ef","Type":"ContainerStarted","Data":"306516ccf14e4e0d155100ecc98c8554a3407d803571bea205d70676851946c8"}
Jul 27 16:57:49.899972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.899938    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"]
Jul 27 16:57:49.902301 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.902277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:49.906809 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.906785    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 16:57:49.907092 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.907065    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 16:57:49.908247 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.908224    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 16:57:49.908357 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.908245    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 16:57:49.916347 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:49.916324    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"]
Jul 27 16:57:50.047173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.047141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24269cee-8cb2-49de-87ed-15012d808b67-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.047343 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.047240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24269cee-8cb2-49de-87ed-15012d808b67-tmp\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.047343 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.047272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2d2\" (UniqueName: \"kubernetes.io/projected/24269cee-8cb2-49de-87ed-15012d808b67-kube-api-access-7f2d2\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.148104 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.148074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24269cee-8cb2-49de-87ed-15012d808b67-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.148248 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.148190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24269cee-8cb2-49de-87ed-15012d808b67-tmp\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.148248 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.148222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2d2\" (UniqueName: \"kubernetes.io/projected/24269cee-8cb2-49de-87ed-15012d808b67-kube-api-access-7f2d2\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.148623 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.148597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24269cee-8cb2-49de-87ed-15012d808b67-tmp\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.150859 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.150807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24269cee-8cb2-49de-87ed-15012d808b67-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.162789 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.162763    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2d2\" (UniqueName: \"kubernetes.io/projected/24269cee-8cb2-49de-87ed-15012d808b67-kube-api-access-7f2d2\") pod \"klusterlet-addon-workmgr-b8f8f4dc7-2m64f\" (UID: \"24269cee-8cb2-49de-87ed-15012d808b67\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.213588 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.213554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:50.571613 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.571519    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"]
Jul 27 16:57:50.573967 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:50.573943    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24269cee_8cb2_49de_87ed_15012d808b67.slice/crio-3efc6701e973df161ada0a6ba5ee19c4b8c0a77a3c6f05b1514b0c1260421173 WatchSource:0}: Error finding container 3efc6701e973df161ada0a6ba5ee19c4b8c0a77a3c6f05b1514b0c1260421173: Status 404 returned error can't find the container with id 3efc6701e973df161ada0a6ba5ee19c4b8c0a77a3c6f05b1514b0c1260421173
Jul 27 16:57:50.986070 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.986031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2" event={"ID":"6ca26a8a-542c-40f6-9af6-d06625d47632","Type":"ContainerStarted","Data":"4cdcb855f073df78db94612ec3caf28ea97b2740ae98665e683ac063cb869d76"}
Jul 27 16:57:50.987753 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.987726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs" event={"ID":"771d4faa-ca73-4478-ac31-1ee76f3ef5ef","Type":"ContainerStarted","Data":"3fe66381a87226e9e117d43e706081ee7923e270264f74f6c305ff2793025c00"}
Jul 27 16:57:50.987923 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.987778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs" event={"ID":"771d4faa-ca73-4478-ac31-1ee76f3ef5ef","Type":"ContainerStarted","Data":"f594a48a6d9ad6e6cafac716428f835a2dc1cf60581bf8f9b7cf43c842965318"}
Jul 27 16:57:50.988960 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.988928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f" event={"ID":"24269cee-8cb2-49de-87ed-15012d808b67","Type":"ContainerStarted","Data":"3efc6701e973df161ada0a6ba5ee19c4b8c0a77a3c6f05b1514b0c1260421173"}
Jul 27 16:57:50.990269 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:50.990246    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"]
Jul 27 16:57:51.007318 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.007302    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"]
Jul 27 16:57:51.007403 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.007372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:51.009349 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.009308    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kt6d2" podStartSLOduration=33.219882155 podStartE2EDuration="36.009293533s" totalImagesPullingTime="2.789411378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:51.008375371 +0000 UTC m=+90.840740369" watchObservedRunningTime="2026-07-27 16:57:51.009293533 +0000 UTC m=+90.841658536"
Jul 27 16:57:51.010794 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.010779    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7dqpm\""
Jul 27 16:57:51.011131 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.011115    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 16:57:51.034741 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.034704    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-mf6bs" podStartSLOduration=32.902388244 podStartE2EDuration="36.034689534s" totalImagesPullingTime="3.13230129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:51.03386749 +0000 UTC m=+90.866232466" watchObservedRunningTime="2026-07-27 16:57:51.034689534 +0000 UTC m=+90.867054513"
Jul 27 16:57:51.156061 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.156037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b9804e1-6ef0-4a2f-91c7-feef292925d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7ft5s\" (UID: \"6b9804e1-6ef0-4a2f-91c7-feef292925d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:51.256863 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.256809    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b9804e1-6ef0-4a2f-91c7-feef292925d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7ft5s\" (UID: \"6b9804e1-6ef0-4a2f-91c7-feef292925d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:51.259053 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.259036    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6b9804e1-6ef0-4a2f-91c7-feef292925d2-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-7ft5s\" (UID: \"6b9804e1-6ef0-4a2f-91c7-feef292925d2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:51.317314 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.317294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:51.434745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.434720    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"]
Jul 27 16:57:51.437586 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:51.437561    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9804e1_6ef0_4a2f_91c7_feef292925d2.slice/crio-a23f8f6644ce0b3d6331df21387aeec7a9611c990e87e1f7ad60682b2760263f WatchSource:0}: Error finding container a23f8f6644ce0b3d6331df21387aeec7a9611c990e87e1f7ad60682b2760263f: Status 404 returned error can't find the container with id a23f8f6644ce0b3d6331df21387aeec7a9611c990e87e1f7ad60682b2760263f
Jul 27 16:57:51.994068 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:51.994026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s" event={"ID":"6b9804e1-6ef0-4a2f-91c7-feef292925d2","Type":"ContainerStarted","Data":"a23f8f6644ce0b3d6331df21387aeec7a9611c990e87e1f7ad60682b2760263f"}
Jul 27 16:57:52.998717 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:52.998680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s" event={"ID":"6b9804e1-6ef0-4a2f-91c7-feef292925d2","Type":"ContainerStarted","Data":"d0e4995f1a14958f9b831fda5d876b08f081debaee287a4928811d68e9bf6104"}
Jul 27 16:57:52.999159 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:52.998917    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:53.004144 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:53.004121    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s"
Jul 27 16:57:53.018769 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:53.018724    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-7ft5s" podStartSLOduration=1.8529291369999998 podStartE2EDuration="3.018708971s" totalImagesPullingTime="1.165779834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:53.018019297 +0000 UTC m=+92.850384273" watchObservedRunningTime="2026-07-27 16:57:53.018708971 +0000 UTC m=+92.851073944"
Jul 27 16:57:54.182518 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.182487    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"]
Jul 27 16:57:54.184576 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.184560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.191166 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.191145    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 16:57:54.191273 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.191207    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x2hxf\""
Jul 27 16:57:54.191350 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.191284    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 16:57:54.191350 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.191303    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 16:57:54.203597 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.203576    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"]
Jul 27 16:57:54.280770 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.280750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.280888 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.280790    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j65tv\" (UniqueName: \"kubernetes.io/projected/4e1d4173-de1d-4976-abfa-1905ffc4333e-kube-api-access-j65tv\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.280888 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.280816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d4173-de1d-4976-abfa-1905ffc4333e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.280888 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.280863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.381412 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.381387    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j65tv\" (UniqueName: \"kubernetes.io/projected/4e1d4173-de1d-4976-abfa-1905ffc4333e-kube-api-access-j65tv\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.381528 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.381423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d4173-de1d-4976-abfa-1905ffc4333e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.381528 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.381448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.381621 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.381599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.382067 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.382049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d4173-de1d-4976-abfa-1905ffc4333e-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.383779 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.383755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.383893 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.383839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e1d4173-de1d-4976-abfa-1905ffc4333e-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.392311 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.392288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j65tv\" (UniqueName: \"kubernetes.io/projected/4e1d4173-de1d-4976-abfa-1905ffc4333e-kube-api-access-j65tv\") pod \"prometheus-operator-84d69f794d-kmnwq\" (UID: \"4e1d4173-de1d-4976-abfa-1905ffc4333e\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.482893 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.482811    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:54.485189 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.485167    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c95f73ba-bcd4-46d1-a5d1-6cb8a4300188-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mkt7j\" (UID: \"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188\") " pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:54.493256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.493231    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"
Jul 27 16:57:54.625555 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.625531    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kmnwq"]
Jul 27 16:57:54.626584 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:54.626562    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1d4173_de1d_4976_abfa_1905ffc4333e.slice/crio-3436d21a97e4751aa1e053125270fbbbd39cde8aeb1a8e0949922c0757efbc97 WatchSource:0}: Error finding container 3436d21a97e4751aa1e053125270fbbbd39cde8aeb1a8e0949922c0757efbc97: Status 404 returned error can't find the container with id 3436d21a97e4751aa1e053125270fbbbd39cde8aeb1a8e0949922c0757efbc97
Jul 27 16:57:54.719341 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.719315    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mkt7j"
Jul 27 16:57:54.857716 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:54.857691    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mkt7j"]
Jul 27 16:57:54.859727 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:57:54.859697    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95f73ba_bcd4_46d1_a5d1_6cb8a4300188.slice/crio-d6aca634660926465bb6153921849814124c84d1d31201460eeadf50a4817ec9 WatchSource:0}: Error finding container d6aca634660926465bb6153921849814124c84d1d31201460eeadf50a4817ec9: Status 404 returned error can't find the container with id d6aca634660926465bb6153921849814124c84d1d31201460eeadf50a4817ec9
Jul 27 16:57:55.006399 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:55.006363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq" event={"ID":"4e1d4173-de1d-4976-abfa-1905ffc4333e","Type":"ContainerStarted","Data":"3436d21a97e4751aa1e053125270fbbbd39cde8aeb1a8e0949922c0757efbc97"}
Jul 27 16:57:55.007512 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:55.007480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkt7j" event={"ID":"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188","Type":"ContainerStarted","Data":"d6aca634660926465bb6153921849814124c84d1d31201460eeadf50a4817ec9"}
Jul 27 16:57:57.018086 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:57.018052    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f" event={"ID":"24269cee-8cb2-49de-87ed-15012d808b67","Type":"ContainerStarted","Data":"8b9ad53afe29705683f658691d413433f0d298eab4082bc994d33b46268e188d"}
Jul 27 16:57:57.019945 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:57.019918    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:57.021544 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:57.021525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f"
Jul 27 16:57:57.041524 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:57.041467    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8f8f4dc7-2m64f" podStartSLOduration=2.3688929 podStartE2EDuration="8.041447602s" totalImagesPullingTime="5.672554702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:57.039863824 +0000 UTC m=+96.872228801" watchObservedRunningTime="2026-07-27 16:57:57.041447602 +0000 UTC m=+96.873812580"
Jul 27 16:57:58.023481 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.023444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq" event={"ID":"4e1d4173-de1d-4976-abfa-1905ffc4333e","Type":"ContainerStarted","Data":"074b459c60669be79c0ab944450ac3a04a60ea62015f7325a10e1fd88f32f0c4"}
Jul 27 16:57:58.023944 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.023536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq" event={"ID":"4e1d4173-de1d-4976-abfa-1905ffc4333e","Type":"ContainerStarted","Data":"70eaf014f42f2dbfb6e0a2eb882f8da78388014234ef2b3bdbfbe0948b8805f3"}
Jul 27 16:57:58.051174 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.051122    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-kmnwq" podStartSLOduration=1.369810979 podStartE2EDuration="4.051108307s" totalImagesPullingTime="2.681297328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:57:58.050847434 +0000 UTC m=+97.883212411" watchObservedRunningTime="2026-07-27 16:57:58.051108307 +0000 UTC m=+97.883473277"
Jul 27 16:57:58.516307 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.516260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:58.516477 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.516326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:58.519125 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.519096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f-metrics-tls\") pod \"dns-default-86hzs\" (UID: \"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f\") " pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:58.519256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.519100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"image-registry-579689bb78-x6msx\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") " pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:58.652843 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.652815    2564 scope.go:117] "RemoveContainer" containerID="f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe"
Jul 27 16:57:58.653039 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:58.653006    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-5w7vz_openshift-console-operator(30c41d72-0d82-4f03-a4bd-19dd758244ae)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podUID="30c41d72-0d82-4f03-a4bd-19dd758244ae"
Jul 27 16:57:58.792092 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.792012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh2hl\""
Jul 27 16:57:58.798827 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.798804    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:57:58.817029 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.817005    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkv78\""
Jul 27 16:57:58.824138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:58.824115    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86hzs"
Jul 27 16:57:59.628921 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.628894    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ppmn9"]
Jul 27 16:57:59.639239 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.639201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.642585 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.642265    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 16:57:59.642585 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.642349    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 16:57:59.642585 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.642368    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qbmgp\""
Jul 27 16:57:59.642806 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.642638    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 16:57:59.724774 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-metrics-client-ca\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.724950 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgplx\" (UniqueName: \"kubernetes.io/projected/513eb75f-69f6-4367-897d-9dad88c76bdf-kube-api-access-jgplx\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.724950 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-textfile\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.724950 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-root\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.724950 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-wtmp\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.724950 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-sys\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.725127 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.724982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.725127 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.725008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.725127 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.725022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826110 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-textfile\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826110 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-root\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826349 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-root\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826349 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-wtmp\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826349 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-sys\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826349 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826347    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826377    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-wtmp\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-textfile\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826411    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513eb75f-69f6-4367-897d-9dad88c76bdf-sys\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826507 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:59.826498    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 16:57:59.826871 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826563    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-metrics-client-ca\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.826871 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:57:59.826586    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls podName:513eb75f-69f6-4367-897d-9dad88c76bdf nodeName:}" failed. No retries permitted until 2026-07-27 16:58:00.326564205 +0000 UTC m=+100.158929173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls") pod "node-exporter-ppmn9" (UID: "513eb75f-69f6-4367-897d-9dad88c76bdf") : secret "node-exporter-tls" not found
Jul 27 16:57:59.826871 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgplx\" (UniqueName: \"kubernetes.io/projected/513eb75f-69f6-4367-897d-9dad88c76bdf-kube-api-access-jgplx\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.827067 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.826984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.827067 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.827046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/513eb75f-69f6-4367-897d-9dad88c76bdf-metrics-client-ca\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.828856 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.828833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:57:59.839363 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:57:59.839316    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgplx\" (UniqueName: \"kubernetes.io/projected/513eb75f-69f6-4367-897d-9dad88c76bdf-kube-api-access-jgplx\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:58:00.330162 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:00.330133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:58:00.332638 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:00.332612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/513eb75f-69f6-4367-897d-9dad88c76bdf-node-exporter-tls\") pod \"node-exporter-ppmn9\" (UID: \"513eb75f-69f6-4367-897d-9dad88c76bdf\") " pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:58:00.551633 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:00.551599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ppmn9"
Jul 27 16:58:00.832351 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:00.832311    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513eb75f_69f6_4367_897d_9dad88c76bdf.slice/crio-d560be03d2a32f9fedea72e710d6eddd03a310510e924b770c7a0bd51e3402c9 WatchSource:0}: Error finding container d560be03d2a32f9fedea72e710d6eddd03a310510e924b770c7a0bd51e3402c9: Status 404 returned error can't find the container with id d560be03d2a32f9fedea72e710d6eddd03a310510e924b770c7a0bd51e3402c9
Jul 27 16:58:00.936276 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:00.936007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgn6f"
Jul 27 16:58:00.980263 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:00.980162    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:58:00.987798 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:00.987771    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9243670c_4cae_4243_a2cb_d58bc418150b.slice/crio-ca3c419eb99b2ef5d1fa951f418621478633e29a52c924c95855a26b4ac85a35 WatchSource:0}: Error finding container ca3c419eb99b2ef5d1fa951f418621478633e29a52c924c95855a26b4ac85a35: Status 404 returned error can't find the container with id ca3c419eb99b2ef5d1fa951f418621478633e29a52c924c95855a26b4ac85a35
Jul 27 16:58:01.008793 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:01.008766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86hzs"]
Jul 27 16:58:01.020216 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:01.020188    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e40ad66_289f_4d3b_bf13_1a1a1ef1f78f.slice/crio-5c44b73aba7ef50e0861512264f377885ee78308fc1bcbcf9b48ed3e57c9874f WatchSource:0}: Error finding container 5c44b73aba7ef50e0861512264f377885ee78308fc1bcbcf9b48ed3e57c9874f: Status 404 returned error can't find the container with id 5c44b73aba7ef50e0861512264f377885ee78308fc1bcbcf9b48ed3e57c9874f
Jul 27 16:58:01.033847 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:01.033821    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hzs" event={"ID":"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f","Type":"ContainerStarted","Data":"5c44b73aba7ef50e0861512264f377885ee78308fc1bcbcf9b48ed3e57c9874f"}
Jul 27 16:58:01.034806 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:01.034782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ppmn9" event={"ID":"513eb75f-69f6-4367-897d-9dad88c76bdf","Type":"ContainerStarted","Data":"d560be03d2a32f9fedea72e710d6eddd03a310510e924b770c7a0bd51e3402c9"}
Jul 27 16:58:01.035850 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:01.035826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579689bb78-x6msx" event={"ID":"9243670c-4cae-4243-a2cb-d58bc418150b","Type":"ContainerStarted","Data":"ca3c419eb99b2ef5d1fa951f418621478633e29a52c924c95855a26b4ac85a35"}
Jul 27 16:58:01.037284 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:01.037261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkt7j" event={"ID":"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188","Type":"ContainerStarted","Data":"d0e84148b3d9e749cb723b9032b1ebf7482945bda2406466a32cfa37a558d542"}
Jul 27 16:58:02.044459 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.044418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579689bb78-x6msx" event={"ID":"9243670c-4cae-4243-a2cb-d58bc418150b","Type":"ContainerStarted","Data":"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"}
Jul 27 16:58:02.044979 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.044933    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:58:02.070964 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.070857    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-579689bb78-x6msx" podStartSLOduration=101.070837937 podStartE2EDuration="1m41.070837937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:02.069123199 +0000 UTC m=+101.901488177" watchObservedRunningTime="2026-07-27 16:58:02.070837937 +0000 UTC m=+101.903202914"
Jul 27 16:58:02.641551 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.641517    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-867779b548-7hf7r"]
Jul 27 16:58:02.657730 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.657703    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.661621 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.661591    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 16:58:02.661746 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.661675    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-smzhp\""
Jul 27 16:58:02.661827 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.661804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 16:58:02.662222 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.662202    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 16:58:02.663049 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.663034    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2nsot81sgpgi8\""
Jul 27 16:58:02.663147 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.663067    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 16:58:02.666234 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.666190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 16:58:02.669260 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.669191    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867779b548-7hf7r"]
Jul 27 16:58:02.751728 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751693    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.751860 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.751860 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751801    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d349f194-1f47-4292-9b5e-d8377de5b469-metrics-client-ca\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.751860 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.752046 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751864    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.752046 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.751909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-grpc-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.752046 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.752025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.752190 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.752067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vcw\" (UniqueName: \"kubernetes.io/projected/d349f194-1f47-4292-9b5e-d8377de5b469-kube-api-access-k7vcw\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.852954 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.852910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853109 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.852958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-grpc-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853273 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vcw\" (UniqueName: \"kubernetes.io/projected/d349f194-1f47-4292-9b5e-d8377de5b469-kube-api-access-k7vcw\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853273 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853370 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.853370 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.853332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d349f194-1f47-4292-9b5e-d8377de5b469-metrics-client-ca\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.854168 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.854117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d349f194-1f47-4292-9b5e-d8377de5b469-metrics-client-ca\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.855975 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.855939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.856256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.856237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.856256 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.856243    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.856344 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.856328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.856587 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.856564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.856773 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.856748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d349f194-1f47-4292-9b5e-d8377de5b469-secret-grpc-tls\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.874541 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.874520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vcw\" (UniqueName: \"kubernetes.io/projected/d349f194-1f47-4292-9b5e-d8377de5b469-kube-api-access-k7vcw\") pod \"thanos-querier-867779b548-7hf7r\" (UID: \"d349f194-1f47-4292-9b5e-d8377de5b469\") " pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:02.969218 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:02.969191    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:05.545239 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.545143    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867779b548-7hf7r"]
Jul 27 16:58:05.755794 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:05.755770    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd349f194_1f47_4292_9b5e_d8377de5b469.slice/crio-381ca05df2ea85c4216bc94a9f7cbdee44020a4a0a5f03597ed70030f2f8b5ec WatchSource:0}: Error finding container 381ca05df2ea85c4216bc94a9f7cbdee44020a4a0a5f03597ed70030f2f8b5ec: Status 404 returned error can't find the container with id 381ca05df2ea85c4216bc94a9f7cbdee44020a4a0a5f03597ed70030f2f8b5ec
Jul 27 16:58:05.851604 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.851548    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:58:05.855437 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.855418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.863754 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.863729    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 16:58:05.864026 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.864009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 16:58:05.865843 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.865820    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 16:58:05.865986 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.865964    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 16:58:05.866093 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866047    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 16:58:05.866501 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866241    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 16:58:05.866501 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866325    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 16:58:05.866501 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866432    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 16:58:05.866501 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866487    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 16:58:05.866728 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866522    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 16:58:05.866728 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866614    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-435833fc529ko\""
Jul 27 16:58:05.866728 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866661    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 16:58:05.866986 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866759    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 16:58:05.866986 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.866800    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hzxnq\""
Jul 27 16:58:05.867207 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.867106    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 16:58:05.869304 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.869030    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 16:58:05.873764 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.873747    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 16:58:05.875804 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.875781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.875891 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.875833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.875945 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.875916    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.875987 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.875957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876024 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.875998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876061 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876061 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876121 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876162 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876116    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876208 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876208 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876277 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876317 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876317 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876408 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876552 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s96sr\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876552 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876504    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.876552 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.876528    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.878137 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.878116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:58:05.977796 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.977766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.977796 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.977798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.977968 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.977919    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978013 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.977982    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978051 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978051 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978158 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978158 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978108    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978158 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.978565 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978375    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.978999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.979214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s96sr\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.980632 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.980312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981375 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.980823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981375 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981375 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981533 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981533 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.981745 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982015 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981965    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982015 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.981988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982173 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.982128    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982767 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.982747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982840 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.982778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.982840 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.982794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.983206 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.983188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.983606 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.983585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.983647 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.983608    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.984198 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.984178    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:05.990009 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:05.989988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s96sr\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr\") pod \"prometheus-k8s-0\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:06.061768 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.061703    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkt7j" event={"ID":"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188","Type":"ContainerStarted","Data":"a56ec7ae11fda9fda945b8540f9f71129a94b38e6dca011daedaa07165cbf781"}
Jul 27 16:58:06.064129 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.063642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hzs" event={"ID":"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f","Type":"ContainerStarted","Data":"d66a29697f276182ded1ed5355ba12174df92adc661c4e3f033e5442e5c7db6d"}
Jul 27 16:58:06.064129 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.063699    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86hzs" event={"ID":"3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f","Type":"ContainerStarted","Data":"e92f1c85c60dcb4715483bd919b8dcc2cec520eac71401b2fc220edb80c68861"}
Jul 27 16:58:06.064129 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.063907    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-86hzs"
Jul 27 16:58:06.065326 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.065303    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"381ca05df2ea85c4216bc94a9f7cbdee44020a4a0a5f03597ed70030f2f8b5ec"}
Jul 27 16:58:06.066854 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.066825    2564 generic.go:356] "Generic (PLEG): container finished" podID="513eb75f-69f6-4367-897d-9dad88c76bdf" containerID="41f4ba71093d5484c6f1ad8002fe13a1965335c3c6219281975e38ad7961ecae" exitCode=0
Jul 27 16:58:06.066971 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.066857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ppmn9" event={"ID":"513eb75f-69f6-4367-897d-9dad88c76bdf","Type":"ContainerDied","Data":"41f4ba71093d5484c6f1ad8002fe13a1965335c3c6219281975e38ad7961ecae"}
Jul 27 16:58:06.087117 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.087080    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-86hzs" podStartSLOduration=67.701825784 podStartE2EDuration="1m12.087066567s" totalImagesPullingTime="4.385240783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:06.086733068 +0000 UTC m=+105.919098044" watchObservedRunningTime="2026-07-27 16:58:06.087066567 +0000 UTC m=+105.919431542"
Jul 27 16:58:06.173530 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.173504    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:06.330849 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:06.330778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:58:06.334151 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:06.334125    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb0a4a3_07b9_4720_8690_2aac1ad1eb18.slice/crio-393f8053287bea8385765e48b012c3cc5de8d5eb33f24440608e1d59b016ab36 WatchSource:0}: Error finding container 393f8053287bea8385765e48b012c3cc5de8d5eb33f24440608e1d59b016ab36: Status 404 returned error can't find the container with id 393f8053287bea8385765e48b012c3cc5de8d5eb33f24440608e1d59b016ab36
Jul 27 16:58:07.081237 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:07.081196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ppmn9" event={"ID":"513eb75f-69f6-4367-897d-9dad88c76bdf","Type":"ContainerStarted","Data":"6b60aab686582703e0d566cb379252effe8f2ddb802f3e93b05b5a70a7a7b90d"}
Jul 27 16:58:07.081696 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:07.081261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ppmn9" event={"ID":"513eb75f-69f6-4367-897d-9dad88c76bdf","Type":"ContainerStarted","Data":"383e338fe680233371f4260e7f16acfd22baece7b30dc411ea3da195cdcd9c30"}
Jul 27 16:58:07.084243 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:07.084216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"393f8053287bea8385765e48b012c3cc5de8d5eb33f24440608e1d59b016ab36"}
Jul 27 16:58:07.103814 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:07.103068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ppmn9" podStartSLOduration=3.065161888 podStartE2EDuration="8.103047996s" totalImagesPullingTime="4.954052361s" totalInitContainerRuntime="83.833747ms" isStatefulPod=true podCreationTimestamp="2026-07-27 16:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:07.102565153 +0000 UTC m=+106.934930130" watchObservedRunningTime="2026-07-27 16:58:07.103047996 +0000 UTC m=+106.935412973"
Jul 27 16:58:08.090710 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:08.090566    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="481633758f29c2bb47fda875491f0733300e8a4d07c7a351ba02e3c245ad4043" exitCode=0
Jul 27 16:58:08.090710 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:08.090656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"481633758f29c2bb47fda875491f0733300e8a4d07c7a351ba02e3c245ad4043"}
Jul 27 16:58:09.098501 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:09.098463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"a0c508d046dfd5465c181515bc42a2b17d49cdb8a60363f0de24f40f9685baae"}
Jul 27 16:58:09.098969 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:09.098535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"657941d4d0c26ca47d6810cfd620903419c64320cdfbc3fb8ccc1ada65464a9b"}
Jul 27 16:58:09.098969 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:09.098570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"520bfe90085bcd3fa557cc94e34f5c99448b373e52ffd559ffbc694f9928b136"}
Jul 27 16:58:12.114605 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.114567    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"6148da8156da15a5091afbab1d87d69865f669d783b8fdcf532790167f02d295"}
Jul 27 16:58:12.115135 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.114821    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:12.115135 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.114911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"95fa328f38a3871d6ccd15b83e942bd7a0eda96d02ab5c0f293e03b692edbef9"}
Jul 27 16:58:12.115135 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.114962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" event={"ID":"d349f194-1f47-4292-9b5e-d8377de5b469","Type":"ContainerStarted","Data":"9999b342f8a6046c5fc226c1dd1c932214087b0f20f601474a7db4b711df02e5"}
Jul 27 16:58:12.116647 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.116624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mkt7j" event={"ID":"c95f73ba-bcd4-46d1-a5d1-6cb8a4300188","Type":"ContainerStarted","Data":"a086469852b63473e726989879322181cdd99a950dad1dad8c86d27710adcc68"}
Jul 27 16:58:12.119625 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119604    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"a6327ed065503abd6cbde2614ba45ae9f07db300928d40b0a902a324348364bb"}
Jul 27 16:58:12.119752 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"5ffa9e3602835cb9f9831f79dc64f7d6d30ad1bb0b35c7cacee93c43580a3469"}
Jul 27 16:58:12.119752 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"1759d4a1a2f62c9ad34f978c9db1c48ce507126beb7e96e8535d5ee5ea47b97e"}
Jul 27 16:58:12.119894 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119758    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"09143b7a040515ce6c7197e03cc18fd8eca0685377c307d3e80bcb8c9fecd88c"}
Jul 27 16:58:12.119894 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"c65487f733035fb1bf7574f6b3a55286a5452028554d61c49a919f3f713616d5"}
Jul 27 16:58:12.119894 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.119845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerStarted","Data":"1007e8b710dc79c4d289a465eee6129a359eccd6ca0d79383ae55710e1d72b1d"}
Jul 27 16:58:12.164079 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.164029    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r" podStartSLOduration=4.513802958 podStartE2EDuration="10.164015231s" totalImagesPullingTime="5.650212273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:12.161232919 +0000 UTC m=+111.993597889" watchObservedRunningTime="2026-07-27 16:58:12.164015231 +0000 UTC m=+111.996380208"
Jul 27 16:58:12.217610 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.217520    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mkt7j" podStartSLOduration=33.687994224 podStartE2EDuration="50.217507564s" totalImagesPullingTime="16.52951334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:57:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:12.216749132 +0000 UTC m=+112.049114120" watchObservedRunningTime="2026-07-27 16:58:12.217507564 +0000 UTC m=+112.049872540"
Jul 27 16:58:12.343698 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:12.343638    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7363366879999997 podStartE2EDuration="7.34362191s" totalImagesPullingTime="4.580508005s" totalInitContainerRuntime="26.777217ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:12.342441829 +0000 UTC m=+112.174806817" watchObservedRunningTime="2026-07-27 16:58:12.34362191 +0000 UTC m=+112.175986936"
Jul 27 16:58:13.129039 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:13.129012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-867779b548-7hf7r"
Jul 27 16:58:13.625337 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:13.625296    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:58:13.653144 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:13.653117    2564 scope.go:117] "RemoveContainer" containerID="f1fecff6069e6ca4efdaff5a0d990c102d222874847c264fad1066e8e8d7c0fe"
Jul 27 16:58:14.129694 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:14.129663    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 16:58:14.130765 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:14.130736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" event={"ID":"30c41d72-0d82-4f03-a4bd-19dd758244ae","Type":"ContainerStarted","Data":"2032d01794925c4ab83e94d8039c896ff45f9f49d233ac6438c5bc4a3ff46bf3"}
Jul 27 16:58:14.131587 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:14.131565    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:58:14.162505 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:14.162457    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz" podStartSLOduration=57.049634933 podStartE2EDuration="59.162442122s" totalImagesPullingTime="2.112807189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:14.160830055 +0000 UTC m=+113.993195032" watchObservedRunningTime="2026-07-27 16:58:14.162442122 +0000 UTC m=+113.994807099"
Jul 27 16:58:14.412823 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:14.412792    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-5w7vz"
Jul 27 16:58:16.086980 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:16.086951    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-86hzs"
Jul 27 16:58:16.174397 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:16.174368    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:58:23.630661 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:23.630630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:58:29.472746 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:29.472710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:58:29.475042 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:29.475016    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35d96c01-af44-46fd-a9fb-d8ff6270d875-metrics-certs\") pod \"network-metrics-daemon-w54gf\" (UID: \"35d96c01-af44-46fd-a9fb-d8ff6270d875\") " pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:58:29.774017 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:29.773950    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7rsp\""
Jul 27 16:58:29.781413 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:29.781392    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w54gf"
Jul 27 16:58:29.949649 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:29.949612    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w54gf"]
Jul 27 16:58:29.951686 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:58:29.951661    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d96c01_af44_46fd_a9fb_d8ff6270d875.slice/crio-2eb699e691d03ce6c3841da1eaf5adf57fcdd37df18cb58cf596f5005f836f76 WatchSource:0}: Error finding container 2eb699e691d03ce6c3841da1eaf5adf57fcdd37df18cb58cf596f5005f836f76: Status 404 returned error can't find the container with id 2eb699e691d03ce6c3841da1eaf5adf57fcdd37df18cb58cf596f5005f836f76
Jul 27 16:58:30.199686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:30.199660    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w54gf" event={"ID":"35d96c01-af44-46fd-a9fb-d8ff6270d875","Type":"ContainerStarted","Data":"2eb699e691d03ce6c3841da1eaf5adf57fcdd37df18cb58cf596f5005f836f76"}
Jul 27 16:58:32.209419 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:32.209384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w54gf" event={"ID":"35d96c01-af44-46fd-a9fb-d8ff6270d875","Type":"ContainerStarted","Data":"2b77915b57b9627877053d145a09b40249dcaad3954ccfab310547a90baa2c4d"}
Jul 27 16:58:32.209869 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:32.209438    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w54gf" event={"ID":"35d96c01-af44-46fd-a9fb-d8ff6270d875","Type":"ContainerStarted","Data":"826bd38a1f540a280b856935d219a3d3f4b8b5a80cd5d19adc89be87a9c44b9b"}
Jul 27 16:58:32.230373 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:32.230326    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w54gf" podStartSLOduration=129.829248866 podStartE2EDuration="2m11.23031079s" totalImagesPullingTime="1.401061924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:32.227917189 +0000 UTC m=+132.060282190" watchObservedRunningTime="2026-07-27 16:58:32.23031079 +0000 UTC m=+132.062675765"
Jul 27 16:58:38.646739 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:38.646699    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-579689bb78-x6msx" podUID="9243670c-4cae-4243-a2cb-d58bc418150b" containerName="registry" containerID="cri-o://789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525" gracePeriod=30
Jul 27 16:58:38.889134 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:38.889112    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:58:39.050830 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050752    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.050830 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050785    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.050830 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frqz9\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051111 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050841    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051111 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051111 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.050954    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051111 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.051000    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051111 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.051028    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token\") pod \"9243670c-4cae-4243-a2cb-d58bc418150b\" (UID: \"9243670c-4cae-4243-a2cb-d58bc418150b\") "
Jul 27 16:58:39.051659 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.051611    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:58:39.051953 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.051921    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:58:39.053414 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.053381    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:58:39.053616 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.053585    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:58:39.053726 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.053581    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:39.053726 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.053642    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9" (OuterVolumeSpecName: "kube-api-access-frqz9") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "kube-api-access-frqz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:39.053726 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.053641    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:58:39.059754 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.059730    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9243670c-4cae-4243-a2cb-d58bc418150b" (UID: "9243670c-4cae-4243-a2cb-d58bc418150b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:58:39.152034 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152012    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-registry-certificates\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152034 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152034    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-bound-sa-token\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152043    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-installation-pull-secrets\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152053    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9243670c-4cae-4243-a2cb-d58bc418150b-image-registry-private-configuration\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152062    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqz9\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-kube-api-access-frqz9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152070    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9243670c-4cae-4243-a2cb-d58bc418150b-ca-trust-extracted\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152079    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9243670c-4cae-4243-a2cb-d58bc418150b-registry-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.152138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.152087    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9243670c-4cae-4243-a2cb-d58bc418150b-trusted-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:58:39.239258 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.239229    2564 generic.go:356] "Generic (PLEG): container finished" podID="9243670c-4cae-4243-a2cb-d58bc418150b" containerID="789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525" exitCode=0
Jul 27 16:58:39.239336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.239278    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-579689bb78-x6msx"
Jul 27 16:58:39.239336 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.239305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579689bb78-x6msx" event={"ID":"9243670c-4cae-4243-a2cb-d58bc418150b","Type":"ContainerDied","Data":"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"}
Jul 27 16:58:39.239405 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.239344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-579689bb78-x6msx" event={"ID":"9243670c-4cae-4243-a2cb-d58bc418150b","Type":"ContainerDied","Data":"ca3c419eb99b2ef5d1fa951f418621478633e29a52c924c95855a26b4ac85a35"}
Jul 27 16:58:39.239405 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.239364    2564 scope.go:117] "RemoveContainer" containerID="789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"
Jul 27 16:58:39.247523 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.247506    2564 scope.go:117] "RemoveContainer" containerID="789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"
Jul 27 16:58:39.247801 ip-10-0-129-84 kubenswrapper[2564]: E0727 16:58:39.247781    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525\": container with ID starting with 789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525 not found: ID does not exist" containerID="789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"
Jul 27 16:58:39.247861 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.247808    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525"} err="failed to get container status \"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525\": rpc error: code = NotFound desc = could not find container \"789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525\": container with ID starting with 789fb690a8a743e1c9d20631dda1dcd8e50a8f99249aac2f69ebc6679c704525 not found: ID does not exist"
Jul 27 16:58:39.285143 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.285123    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:58:39.293293 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.293277    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-579689bb78-x6msx"]
Jul 27 16:58:39.757586 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:39.757554    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-kt6d2_6ca26a8a-542c-40f6-9af6-d06625d47632/cluster-monitoring-operator/0.log"
Jul 27 16:58:40.657427 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:40.657393    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9243670c-4cae-4243-a2cb-d58bc418150b" path="/var/lib/kubelet/pods/9243670c-4cae-4243-a2cb-d58bc418150b/volumes"
Jul 27 16:58:40.954744 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:40.954676    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/init-textfile/0.log"
Jul 27 16:58:41.155925 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:41.155898    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/node-exporter/0.log"
Jul 27 16:58:41.355120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:41.355041    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/kube-rbac-proxy/0.log"
Jul 27 16:58:43.355322 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:43.355296    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/init-config-reloader/0.log"
Jul 27 16:58:43.557200 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:43.557169    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/prometheus/0.log"
Jul 27 16:58:43.755776 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:43.755746    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/config-reloader/0.log"
Jul 27 16:58:43.955119 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:43.955091    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/thanos-sidecar/0.log"
Jul 27 16:58:44.155073 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:44.155048    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/kube-rbac-proxy-web/0.log"
Jul 27 16:58:44.354934 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:44.354902    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/kube-rbac-proxy/0.log"
Jul 27 16:58:44.555449 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:44.555379    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/kube-rbac-proxy-thanos/0.log"
Jul 27 16:58:44.757138 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:44.757110    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kmnwq_4e1d4173-de1d-4976-abfa-1905ffc4333e/prometheus-operator/0.log"
Jul 27 16:58:44.956526 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:44.956494    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kmnwq_4e1d4173-de1d-4976-abfa-1905ffc4333e/kube-rbac-proxy/0.log"
Jul 27 16:58:45.155577 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:45.155546    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-7ft5s_6b9804e1-6ef0-4a2f-91c7-feef292925d2/prometheus-operator-admission-webhook/0.log"
Jul 27 16:58:45.354545 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:45.354461    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/thanos-query/0.log"
Jul 27 16:58:45.554474 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:45.554445    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-web/0.log"
Jul 27 16:58:45.754485 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:45.754464    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy/0.log"
Jul 27 16:58:45.954580 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:45.954549    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/prom-label-proxy/0.log"
Jul 27 16:58:46.154321 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:46.154296    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-rules/0.log"
Jul 27 16:58:46.355061 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:46.355033    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-metrics/0.log"
Jul 27 16:58:46.754428 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:46.754401    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 16:58:46.957133 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:58:46.957103    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/3.log"
Jul 27 16:59:06.174584 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:06.174553    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:06.194947 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:06.194919    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:06.361801 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:06.361777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:24.274473 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274441    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274729    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="prometheus" containerID="cri-o://1007e8b710dc79c4d289a465eee6129a359eccd6ca0d79383ae55710e1d72b1d" gracePeriod=600
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274753    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy" containerID="cri-o://5ffa9e3602835cb9f9831f79dc64f7d6d30ad1bb0b35c7cacee93c43580a3469" gracePeriod=600
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274773    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="thanos-sidecar" containerID="cri-o://09143b7a040515ce6c7197e03cc18fd8eca0685377c307d3e80bcb8c9fecd88c" gracePeriod=600
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274834    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a6327ed065503abd6cbde2614ba45ae9f07db300928d40b0a902a324348364bb" gracePeriod=600
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274825    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="config-reloader" containerID="cri-o://c65487f733035fb1bf7574f6b3a55286a5452028554d61c49a919f3f713616d5" gracePeriod=600
Jul 27 16:59:24.274924 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.274799    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-web" containerID="cri-o://1759d4a1a2f62c9ad34f978c9db1c48ce507126beb7e96e8535d5ee5ea47b97e" gracePeriod=600
Jul 27 16:59:24.417030 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417003    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="a6327ed065503abd6cbde2614ba45ae9f07db300928d40b0a902a324348364bb" exitCode=0
Jul 27 16:59:24.417030 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417026    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="5ffa9e3602835cb9f9831f79dc64f7d6d30ad1bb0b35c7cacee93c43580a3469" exitCode=0
Jul 27 16:59:24.417030 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417034    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="1759d4a1a2f62c9ad34f978c9db1c48ce507126beb7e96e8535d5ee5ea47b97e" exitCode=0
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417039    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="09143b7a040515ce6c7197e03cc18fd8eca0685377c307d3e80bcb8c9fecd88c" exitCode=0
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417045    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="c65487f733035fb1bf7574f6b3a55286a5452028554d61c49a919f3f713616d5" exitCode=0
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417050    2564 generic.go:356] "Generic (PLEG): container finished" podID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerID="1007e8b710dc79c4d289a465eee6129a359eccd6ca0d79383ae55710e1d72b1d" exitCode=0
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"a6327ed065503abd6cbde2614ba45ae9f07db300928d40b0a902a324348364bb"}
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"5ffa9e3602835cb9f9831f79dc64f7d6d30ad1bb0b35c7cacee93c43580a3469"}
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"1759d4a1a2f62c9ad34f978c9db1c48ce507126beb7e96e8535d5ee5ea47b97e"}
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"09143b7a040515ce6c7197e03cc18fd8eca0685377c307d3e80bcb8c9fecd88c"}
Jul 27 16:59:24.417213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"c65487f733035fb1bf7574f6b3a55286a5452028554d61c49a919f3f713616d5"}
Jul 27 16:59:24.417454 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.417236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"1007e8b710dc79c4d289a465eee6129a359eccd6ca0d79383ae55710e1d72b1d"}
Jul 27 16:59:24.513449 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.513149    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:24.584633 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584538    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s96sr\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584633 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584575    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584633 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584599    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584633 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584615    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584651    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584685    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584717    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584741    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584774    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584804    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584900    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.584972 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.584994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585124    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585464 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1\") pod \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\" (UID: \"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18\") "
Jul 27 16:59:24.585759 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.585682    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.588028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.586481    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.588028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.586531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.588028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.586571    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:24.588028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.587655    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.588028 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.587859    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.588379 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.588302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.589088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.588517    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:24.589088 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.588535    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:24.590059 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.589985    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592231    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592285    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config" (OuterVolumeSpecName: "config") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592327    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592357    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out" (OuterVolumeSpecName: "config-out") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592369    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr" (OuterVolumeSpecName: "kube-api-access-s96sr") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "kube-api-access-s96sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:24.592963 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.592432    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.606033 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.606008    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config" (OuterVolumeSpecName: "web-config") pod "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" (UID: "5bb0a4a3-07b9-4720-8690-2aac1ad1eb18"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:59:24.686316 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686296    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-metrics-client-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686316 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686317    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686326    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s96sr\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-kube-api-access-s96sr\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686336    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-web-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686345    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-tls-assets\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686353    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686361    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-db\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686369    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-kube-rbac-proxy\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686377    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config-out\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686386    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686394    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686403    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-metrics-client-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686411    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-grpc-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686419    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-config\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686426    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686433    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686441 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686441    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686855 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686450    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686855 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686458    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:24.686855 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:24.686466    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 16:59:25.423479 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.423449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5bb0a4a3-07b9-4720-8690-2aac1ad1eb18","Type":"ContainerDied","Data":"393f8053287bea8385765e48b012c3cc5de8d5eb33f24440608e1d59b016ab36"}
Jul 27 16:59:25.423479 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.423480    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.423930 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.423524    2564 scope.go:117] "RemoveContainer" containerID="a6327ed065503abd6cbde2614ba45ae9f07db300928d40b0a902a324348364bb"
Jul 27 16:59:25.432428 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.432408    2564 scope.go:117] "RemoveContainer" containerID="5ffa9e3602835cb9f9831f79dc64f7d6d30ad1bb0b35c7cacee93c43580a3469"
Jul 27 16:59:25.439106 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.439089    2564 scope.go:117] "RemoveContainer" containerID="1759d4a1a2f62c9ad34f978c9db1c48ce507126beb7e96e8535d5ee5ea47b97e"
Jul 27 16:59:25.445177 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.445160    2564 scope.go:117] "RemoveContainer" containerID="09143b7a040515ce6c7197e03cc18fd8eca0685377c307d3e80bcb8c9fecd88c"
Jul 27 16:59:25.451250 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.451229    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:25.451626 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.451610    2564 scope.go:117] "RemoveContainer" containerID="c65487f733035fb1bf7574f6b3a55286a5452028554d61c49a919f3f713616d5"
Jul 27 16:59:25.457164 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.457144    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:25.458006 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.457995    2564 scope.go:117] "RemoveContainer" containerID="1007e8b710dc79c4d289a465eee6129a359eccd6ca0d79383ae55710e1d72b1d"
Jul 27 16:59:25.464597 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.464582    2564 scope.go:117] "RemoveContainer" containerID="481633758f29c2bb47fda875491f0733300e8a4d07c7a351ba02e3c245ad4043"
Jul 27 16:59:25.488330 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488311    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:25.488555 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488544    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="thanos-sidecar"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488557    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="thanos-sidecar"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488565    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488570    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488579    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="init-config-reloader"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488585    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="init-config-reloader"
Jul 27 16:59:25.488595 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488593    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="config-reloader"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488598    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="config-reloader"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488607    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="prometheus"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488613    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="prometheus"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488618    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-web"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488623    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-web"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488633    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-thanos"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488638    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-thanos"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488643    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9243670c-4cae-4243-a2cb-d58bc418150b" containerName="registry"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488649    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9243670c-4cae-4243-a2cb-d58bc418150b" containerName="registry"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488690    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="thanos-sidecar"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488697    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="prometheus"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488702    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-thanos"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488708    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="config-reloader"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488714    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488719    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9243670c-4cae-4243-a2cb-d58bc418150b" containerName="registry"
Jul 27 16:59:25.488762 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.488724    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" containerName="kube-rbac-proxy-web"
Jul 27 16:59:25.492280 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.492262    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.495247 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.495230    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 16:59:25.495551 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.495536    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 16:59:25.495638 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.495624    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-435833fc529ko\""
Jul 27 16:59:25.495705 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.495668    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 16:59:25.495891 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.495866    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 16:59:25.496396 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.496382    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 16:59:25.496557 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.496537    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 16:59:25.496837 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.496820    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 16:59:25.496929 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.496841    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 16:59:25.496929 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.496824    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hzxnq\""
Jul 27 16:59:25.497160 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.497142    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 16:59:25.497249 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.497230    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 16:59:25.497543 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.497527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 16:59:25.497624 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.497527    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 16:59:25.497822 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.497805    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 16:59:25.499971 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.499950    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 16:59:25.501900 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.501858    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 16:59:25.513855 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.513833    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:25.592604 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592524    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592604 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592604 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-web-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592736    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgd9\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-kube-api-access-jwgd9\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.592817 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592923    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-config-out\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.592969    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.593029    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593120 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.593069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.593377 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.593135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694081 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694052    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694105    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694213 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-web-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgd9\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-kube-api-access-jwgd9\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694466 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-config-out\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694560    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.694792 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.694661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.695183    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.695576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.695738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.695834    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.696529    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.697036    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.697399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.697826 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.697599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26f8f641-4932-468b-9933-504c96e7e097-config-out\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.698253 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.697942    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.698584 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.698535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/26f8f641-4932-468b-9933-504c96e7e097-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.698702 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.698624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.699409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.699055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.699409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.699326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.699409 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.699368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-web-config\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.699616 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.699433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.699828 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.699807    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.700093 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.700077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.700335 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.700311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.700686 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.700667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26f8f641-4932-468b-9933-504c96e7e097-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.704209 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.704194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgd9\" (UniqueName: \"kubernetes.io/projected/26f8f641-4932-468b-9933-504c96e7e097-kube-api-access-jwgd9\") pod \"prometheus-k8s-0\" (UID: \"26f8f641-4932-468b-9933-504c96e7e097\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.802167 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.802137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:59:25.939475 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:25.939447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:59:25.940692 ip-10-0-129-84 kubenswrapper[2564]: W0727 16:59:25.940670    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f8f641_4932_468b_9933_504c96e7e097.slice/crio-b1bd61dc25927e467a722cc9d06682ef61407f2be7a9b3593a57464f4dc3afce WatchSource:0}: Error finding container b1bd61dc25927e467a722cc9d06682ef61407f2be7a9b3593a57464f4dc3afce: Status 404 returned error can't find the container with id b1bd61dc25927e467a722cc9d06682ef61407f2be7a9b3593a57464f4dc3afce
Jul 27 16:59:26.427948 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:26.427917    2564 generic.go:356] "Generic (PLEG): container finished" podID="26f8f641-4932-468b-9933-504c96e7e097" containerID="4b358ba65a3e5850fa178501eeb44a68b510b17d15e5133fcfdaf7bed1cc6144" exitCode=0
Jul 27 16:59:26.428400 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:26.428013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerDied","Data":"4b358ba65a3e5850fa178501eeb44a68b510b17d15e5133fcfdaf7bed1cc6144"}
Jul 27 16:59:26.428400 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:26.428124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"b1bd61dc25927e467a722cc9d06682ef61407f2be7a9b3593a57464f4dc3afce"}
Jul 27 16:59:26.659146 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:26.659118    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb0a4a3-07b9-4720-8690-2aac1ad1eb18" path="/var/lib/kubelet/pods/5bb0a4a3-07b9-4720-8690-2aac1ad1eb18/volumes"
Jul 27 16:59:27.435960 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.435927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"db01cea033e3c9694737e30b06c8d7becaf9e46a7a564b0d6ccdce68d878bc2c"}
Jul 27 16:59:27.436380 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.436000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"ecdb6a69e5d572f44f01ebfb1e8b62769c753d47603927a21ce48ed5ab08df72"}
Jul 27 16:59:27.436380 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.436041    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"464883792dba9a0b10118a15d15a2e30e255dd50e25ee2376297e3e3e341a3a7"}
Jul 27 16:59:27.436380 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.436081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"97d7a5efb07b31934eab2eae4bdb71eab79fda6fb230b9a8109f02b1bd49b5e0"}
Jul 27 16:59:27.436380 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.436123    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"b995fe7e33212ecf6b9f4ee0c9d98336669e52947253d0ed6d0b70d36a60d171"}
Jul 27 16:59:27.436380 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.436160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"26f8f641-4932-468b-9933-504c96e7e097","Type":"ContainerStarted","Data":"a1491ee49393d05768248aaeb073f37f1ab22a564b9ccb99edfeb88de0cfb834"}
Jul 27 16:59:27.470259 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:27.470207    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.454658392 podStartE2EDuration="2.470191544s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.533152ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:27.467376152 +0000 UTC m=+187.299741128" watchObservedRunningTime="2026-07-27 16:59:27.470191544 +0000 UTC m=+187.302556519"
Jul 27 16:59:30.802430 ip-10-0-129-84 kubenswrapper[2564]: I0727 16:59:30.802399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:00:25.802647 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:00:25.802611    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:00:25.817895 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:00:25.817854    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:00:26.678242 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:00:26.678214    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:01:20.547838 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:01:20.547813    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:01:20.552126 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:01:20.552102    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:01:20.558217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:01:20.558188    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:02:20.816430 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.816392    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rlfgw"]
Jul 27 17:02:20.821791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.819777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.823414 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.823383    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 17:02:20.823583 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.823563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 17:02:20.836557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.836529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rlfgw"]
Jul 27 17:02:20.847677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.847657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfts\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-kube-api-access-zcfts\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.847771 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.847696    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.948356 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.948331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfts\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-kube-api-access-zcfts\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.948483 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.948376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.962372 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.962346    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:20.962571 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:20.962551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfts\" (UniqueName: \"kubernetes.io/projected/8d678d2f-e481-4b88-9f6e-3af76d7cb853-kube-api-access-zcfts\") pod \"cert-manager-webhook-6486ffd997-rlfgw\" (UID: \"8d678d2f-e481-4b88-9f6e-3af76d7cb853\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:21.142737 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:21.142677    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:21.269344 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:21.269318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rlfgw"]
Jul 27 17:02:21.271238 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:02:21.271211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d678d2f_e481_4b88_9f6e_3af76d7cb853.slice/crio-068a1e66e69822af917c75870ca1aa9dce44bf75f8d10de483e2462d1c7308a1 WatchSource:0}: Error finding container 068a1e66e69822af917c75870ca1aa9dce44bf75f8d10de483e2462d1c7308a1: Status 404 returned error can't find the container with id 068a1e66e69822af917c75870ca1aa9dce44bf75f8d10de483e2462d1c7308a1
Jul 27 17:02:21.273442 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:21.273426    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:02:22.069663 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:22.069625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw" event={"ID":"8d678d2f-e481-4b88-9f6e-3af76d7cb853","Type":"ContainerStarted","Data":"068a1e66e69822af917c75870ca1aa9dce44bf75f8d10de483e2462d1c7308a1"}
Jul 27 17:02:26.088061 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:26.088027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw" event={"ID":"8d678d2f-e481-4b88-9f6e-3af76d7cb853","Type":"ContainerStarted","Data":"4774318c1dce83347d12d92bb7f83c788329668d70eae3c07f6af9df614e1e9e"}
Jul 27 17:02:26.088438 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:26.088104    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:26.105919 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:26.105860    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw" podStartSLOduration=2.313870756 podStartE2EDuration="6.105847592s" totalImagesPullingTime="3.791976836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:26.104782017 +0000 UTC m=+365.937146994" watchObservedRunningTime="2026-07-27 17:02:26.105847592 +0000 UTC m=+365.938212568"
Jul 27 17:02:32.094025 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:32.093990    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rlfgw"
Jul 27 17:02:58.696690 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.696663    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"]
Jul 27 17:02:58.701323 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.701308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.704028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.704000    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h676l\""
Jul 27 17:02:58.704158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.704039    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 17:02:58.704397 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.704379    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 17:02:58.704450 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.704400    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 17:02:58.704703 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.704690    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 17:02:58.732356 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.732332    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"]
Jul 27 17:02:58.831435 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.831409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.831588 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.831451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.831588 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.831482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.831588 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.831525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nff5b\" (UniqueName: \"kubernetes.io/projected/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-kube-api-access-nff5b\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.831697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.831598    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932348    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nff5b\" (UniqueName: \"kubernetes.io/projected/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-kube-api-access-nff5b\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932653 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932653 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932436    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.932864 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.932833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-manifests\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.935037 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.935015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-tmp\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.935037 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.935026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.935855 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.935830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-webhook-cert\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:58.941479 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:58.941459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nff5b\" (UniqueName: \"kubernetes.io/projected/7766aaff-8379-4d13-8ff8-c1a38a03d2c4-kube-api-access-nff5b\") pod \"opendatahub-operator-controller-manager-565c484996-dcfj6\" (UID: \"7766aaff-8379-4d13-8ff8-c1a38a03d2c4\") " pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:59.011771 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:59.011746    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:02:59.183308 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:59.183231    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"]
Jul 27 17:02:59.186107 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:02:59.186077    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7766aaff_8379_4d13_8ff8_c1a38a03d2c4.slice/crio-79c45b00a2fafc99e6a8431a8d75d23650653ce57f91238273be2b7b91719089 WatchSource:0}: Error finding container 79c45b00a2fafc99e6a8431a8d75d23650653ce57f91238273be2b7b91719089: Status 404 returned error can't find the container with id 79c45b00a2fafc99e6a8431a8d75d23650653ce57f91238273be2b7b91719089
Jul 27 17:02:59.208070 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:02:59.208047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6" event={"ID":"7766aaff-8379-4d13-8ff8-c1a38a03d2c4","Type":"ContainerStarted","Data":"79c45b00a2fafc99e6a8431a8d75d23650653ce57f91238273be2b7b91719089"}
Jul 27 17:03:04.234284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:04.234240    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6" event={"ID":"7766aaff-8379-4d13-8ff8-c1a38a03d2c4","Type":"ContainerStarted","Data":"710a1595c24f7147def377773bfe00e4849d5eb15201fd71c0ce4c5c88aaccba"}
Jul 27 17:03:05.240427 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:05.240393    2564 generic.go:356] "Generic (PLEG): container finished" podID="7766aaff-8379-4d13-8ff8-c1a38a03d2c4" containerID="710a1595c24f7147def377773bfe00e4849d5eb15201fd71c0ce4c5c88aaccba" exitCode=0
Jul 27 17:03:05.240871 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:05.240477    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6" event={"ID":"7766aaff-8379-4d13-8ff8-c1a38a03d2c4","Type":"ContainerDied","Data":"710a1595c24f7147def377773bfe00e4849d5eb15201fd71c0ce4c5c88aaccba"}
Jul 27 17:03:06.246175 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:06.246141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6" event={"ID":"7766aaff-8379-4d13-8ff8-c1a38a03d2c4","Type":"ContainerStarted","Data":"90b6829bd650d56ebe0ca0722d40fb1802033e7f827ccd6d4130891565b62072"}
Jul 27 17:03:06.246557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:06.246293    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:03:06.277543 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:06.277501    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6" podStartSLOduration=2.791507394 podStartE2EDuration="8.277484959s" totalImagesPullingTime="4.439990485s" totalInitContainerRuntime="1.04598708s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:06.275611731 +0000 UTC m=+406.107976715" watchObservedRunningTime="2026-07-27 17:03:06.277484959 +0000 UTC m=+406.109849937"
Jul 27 17:03:17.252474 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:17.252448    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565c484996-dcfj6"
Jul 27 17:03:18.263015 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.262977    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b56999664-dstbd"]
Jul 27 17:03:18.266508 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.266482    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.270712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270685    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:03:18.270712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270702    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 17:03:18.271158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270837    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gvlzk\""
Jul 27 17:03:18.271158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270897    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 17:03:18.271158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270944    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 17:03:18.271158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.270977    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 17:03:18.276523 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.276503    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b56999664-dstbd"]
Jul 27 17:03:18.382020 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.381997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.382118 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.382044    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5715a615-4359-43f5-a41c-54dd9f6c95b4-manager-config\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.382118 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.382081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-metrics-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.382118 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.382097    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvpxj\" (UniqueName: \"kubernetes.io/projected/5715a615-4359-43f5-a41c-54dd9f6c95b4-kube-api-access-zvpxj\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.482948 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.482928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.483042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.482976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5715a615-4359-43f5-a41c-54dd9f6c95b4-manager-config\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.483042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.483020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-metrics-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.483136 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.483043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvpxj\" (UniqueName: \"kubernetes.io/projected/5715a615-4359-43f5-a41c-54dd9f6c95b4-kube-api-access-zvpxj\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.483689 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.483671    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5715a615-4359-43f5-a41c-54dd9f6c95b4-manager-config\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.485480 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.485457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-metrics-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.485558 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.485506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5715a615-4359-43f5-a41c-54dd9f6c95b4-cert\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.490528 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.490510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvpxj\" (UniqueName: \"kubernetes.io/projected/5715a615-4359-43f5-a41c-54dd9f6c95b4-kube-api-access-zvpxj\") pod \"lws-controller-manager-b56999664-dstbd\" (UID: \"5715a615-4359-43f5-a41c-54dd9f6c95b4\") " pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.576096 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.576046    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:18.697597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:18.697574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b56999664-dstbd"]
Jul 27 17:03:18.699858 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:03:18.699833    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5715a615_4359_43f5_a41c_54dd9f6c95b4.slice/crio-b8d38b4c6b355c2be1154e0da7f6d3bff83df52968bbe8b7cccec711b829401d WatchSource:0}: Error finding container b8d38b4c6b355c2be1154e0da7f6d3bff83df52968bbe8b7cccec711b829401d: Status 404 returned error can't find the container with id b8d38b4c6b355c2be1154e0da7f6d3bff83df52968bbe8b7cccec711b829401d
Jul 27 17:03:19.294896 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:19.294852    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd" event={"ID":"5715a615-4359-43f5-a41c-54dd9f6c95b4","Type":"ContainerStarted","Data":"b8d38b4c6b355c2be1154e0da7f6d3bff83df52968bbe8b7cccec711b829401d"}
Jul 27 17:03:22.308871 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:22.308832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd" event={"ID":"5715a615-4359-43f5-a41c-54dd9f6c95b4","Type":"ContainerStarted","Data":"debf9e73ee7f8cf01446177e70be7ac6b3c7423da9cc23fc2a9ba35005f1b013"}
Jul 27 17:03:22.309264 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:22.309031    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:22.327436 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:22.327393    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd" podStartSLOduration=1.721206193 podStartE2EDuration="4.32737963s" totalImagesPullingTime="2.606173437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:22.326973903 +0000 UTC m=+422.159338898" watchObservedRunningTime="2026-07-27 17:03:22.32737963 +0000 UTC m=+422.159744655"
Jul 27 17:03:31.146155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.146123    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"]
Jul 27 17:03:31.148537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.148521    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.155130 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.154862    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 17:03:31.155215 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.155172    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mdvcg\""
Jul 27 17:03:31.158669 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.158652    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 17:03:31.158790 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.158708    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 17:03:31.158790 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.158720    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 17:03:31.174920 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.174898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tls-certs\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.175018 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.174944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tmp\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.175060 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.175015    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44km\" (UniqueName: \"kubernetes.io/projected/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-kube-api-access-s44km\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.181639 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.181616    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"]
Jul 27 17:03:31.275423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.275401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tmp\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.275531 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.275447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44km\" (UniqueName: \"kubernetes.io/projected/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-kube-api-access-s44km\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.275531 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.275487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tls-certs\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.277486 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.277462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tmp\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.277787 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.277768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-tls-certs\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.294930 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.294911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44km\" (UniqueName: \"kubernetes.io/projected/d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6-kube-api-access-s44km\") pod \"kube-auth-proxy-5cc796b767-jpfpv\" (UID: \"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6\") " pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.458336 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.458275    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"
Jul 27 17:03:31.585267 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:31.585235    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv"]
Jul 27 17:03:31.587019 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:03:31.586995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a7e101_87da_40ce_b3f5_7eaed4b6f0a6.slice/crio-8f6a8dfeec4903e2881cdee3ef35a96e9760a6f8dcaf4a0cc500117a0d203ba4 WatchSource:0}: Error finding container 8f6a8dfeec4903e2881cdee3ef35a96e9760a6f8dcaf4a0cc500117a0d203ba4: Status 404 returned error can't find the container with id 8f6a8dfeec4903e2881cdee3ef35a96e9760a6f8dcaf4a0cc500117a0d203ba4
Jul 27 17:03:32.363015 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:32.358211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv" event={"ID":"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6","Type":"ContainerStarted","Data":"8f6a8dfeec4903e2881cdee3ef35a96e9760a6f8dcaf4a0cc500117a0d203ba4"}
Jul 27 17:03:33.315380 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:33.315354    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b56999664-dstbd"
Jul 27 17:03:35.372714 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:35.372678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv" event={"ID":"d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6","Type":"ContainerStarted","Data":"4953fb2626fc5134c775c01784ef6f8dd5234ad89129500e77e9d836f252f83a"}
Jul 27 17:03:35.395497 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:03:35.395434    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc796b767-jpfpv" podStartSLOduration=1.524821335 podStartE2EDuration="4.395414661s" totalImagesPullingTime="2.870593326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:35.39364988 +0000 UTC m=+435.226014869" watchObservedRunningTime="2026-07-27 17:03:35.395414661 +0000 UTC m=+435.227779640"
Jul 27 17:05:08.049133 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.049102    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"]
Jul 27 17:05:08.051200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.051185    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.054771 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.054739    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rs4tp\""
Jul 27 17:05:08.054928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.054782    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 17:05:08.054928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.054868    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 17:05:08.069202 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.069177    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"]
Jul 27 17:05:08.089177 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.089152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.089281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.089213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwl9\" (UniqueName: \"kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.189576 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.189556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.189664 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.189601    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwl9\" (UniqueName: \"kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.189972 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.189954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.198654 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.198631    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwl9\" (UniqueName: \"kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.360865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.360814    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:08.491869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.490024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"]
Jul 27 17:05:08.493768 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:05:08.493739    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8594521b_0c37_4657_a630_cd40ccb8de11.slice/crio-51158e9bdf6baf527e98e3f8c76c184519dbc045a47722221eefb3339bfd44a5 WatchSource:0}: Error finding container 51158e9bdf6baf527e98e3f8c76c184519dbc045a47722221eefb3339bfd44a5: Status 404 returned error can't find the container with id 51158e9bdf6baf527e98e3f8c76c184519dbc045a47722221eefb3339bfd44a5
Jul 27 17:05:08.760453 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:08.760423    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw" event={"ID":"8594521b-0c37-4657-a630-cd40ccb8de11","Type":"ContainerStarted","Data":"51158e9bdf6baf527e98e3f8c76c184519dbc045a47722221eefb3339bfd44a5"}
Jul 27 17:05:15.792971 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:15.792930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw" event={"ID":"8594521b-0c37-4657-a630-cd40ccb8de11","Type":"ContainerStarted","Data":"750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951"}
Jul 27 17:05:15.793389 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:15.793086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:15.819644 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:15.819600    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw" podStartSLOduration=1.027719644 podStartE2EDuration="7.81958672s" totalImagesPullingTime="6.791867076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:15.819128277 +0000 UTC m=+535.651493254" watchObservedRunningTime="2026-07-27 17:05:15.81958672 +0000 UTC m=+535.651951695"
Jul 27 17:05:26.799794 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:26.799766    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:28.537089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.537051    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"]
Jul 27 17:05:28.537544 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.537287    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" containerName="manager" containerID="cri-o://750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951" gracePeriod=2
Jul 27 17:05:28.563516 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.563490    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"]
Jul 27 17:05:28.595517 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.595487    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:28.595911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.595869    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" containerName="manager"
Jul 27 17:05:28.595911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.595906    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" containerName="manager"
Jul 27 17:05:28.596042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.595967    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" containerName="manager"
Jul 27 17:05:28.597986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.597971    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.600783 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:05:28.600749    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k65vw\" is forbidden: User \"system:node:ip-10-0-129-84.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-84.ec2.internal' and this object" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:28.615972 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.615948    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:28.651048 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.651017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9f89\" (UniqueName: \"kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.651215 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.651060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.751574 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.751537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9f89\" (UniqueName: \"kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.751746 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.751622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.752016 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.751995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.760298 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.760277    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9f89\" (UniqueName: \"kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h44nw\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.776581 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.776561    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:28.848068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.848006    2564 generic.go:356] "Generic (PLEG): container finished" podID="8594521b-0c37-4657-a630-cd40ccb8de11" containerID="750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951" exitCode=0
Jul 27 17:05:28.848068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.848047    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k65vw"
Jul 27 17:05:28.848198 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.848104    2564 scope.go:117] "RemoveContainer" containerID="750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951"
Jul 27 17:05:28.852216 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.852194    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume\") pod \"8594521b-0c37-4657-a630-cd40ccb8de11\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") "
Jul 27 17:05:28.852303 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.852255    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwl9\" (UniqueName: \"kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9\") pod \"8594521b-0c37-4657-a630-cd40ccb8de11\" (UID: \"8594521b-0c37-4657-a630-cd40ccb8de11\") "
Jul 27 17:05:28.852594 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.852574    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8594521b-0c37-4657-a630-cd40ccb8de11" (UID: "8594521b-0c37-4657-a630-cd40ccb8de11"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:05:28.854202 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.854183    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9" (OuterVolumeSpecName: "kube-api-access-5hwl9") pod "8594521b-0c37-4657-a630-cd40ccb8de11" (UID: "8594521b-0c37-4657-a630-cd40ccb8de11"). InnerVolumeSpecName "kube-api-access-5hwl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:05:28.855630 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.855616    2564 scope.go:117] "RemoveContainer" containerID="750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951"
Jul 27 17:05:28.855914 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:05:28.855889    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951\": container with ID starting with 750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951 not found: ID does not exist" containerID="750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951"
Jul 27 17:05:28.856008 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.855920    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951"} err="failed to get container status \"750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951\": rpc error: code = NotFound desc = could not find container \"750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951\": container with ID starting with 750acf41960ebbb1b78cc395cd3a6122dbe4506789be2361e83de77cfbfc4951 not found: ID does not exist"
Jul 27 17:05:28.921256 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.921232    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:28.952901 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.952867    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8594521b-0c37-4657-a630-cd40ccb8de11-extensions-socket-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:05:28.952901 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:28.952902    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwl9\" (UniqueName: \"kubernetes.io/projected/8594521b-0c37-4657-a630-cd40ccb8de11-kube-api-access-5hwl9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:05:29.070452 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:29.070428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:29.073218 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:05:29.073191    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca4694b_d9e7_4829_994e_9525db983307.slice/crio-9a9514b0ebc3c0858ff85037b679c8453c6dd302950ac8429993bfac485d8aa0 WatchSource:0}: Error finding container 9a9514b0ebc3c0858ff85037b679c8453c6dd302950ac8429993bfac485d8aa0: Status 404 returned error can't find the container with id 9a9514b0ebc3c0858ff85037b679c8453c6dd302950ac8429993bfac485d8aa0
Jul 27 17:05:29.853481 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:29.853442    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" event={"ID":"9ca4694b-d9e7-4829-994e-9525db983307","Type":"ContainerStarted","Data":"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"}
Jul 27 17:05:29.853984 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:29.853504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" event={"ID":"9ca4694b-d9e7-4829-994e-9525db983307","Type":"ContainerStarted","Data":"9a9514b0ebc3c0858ff85037b679c8453c6dd302950ac8429993bfac485d8aa0"}
Jul 27 17:05:29.853984 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:29.853530    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:29.879419 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:29.879373    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" podStartSLOduration=1.879355911 podStartE2EDuration="1.879355911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:29.87657656 +0000 UTC m=+549.708941571" watchObservedRunningTime="2026-07-27 17:05:29.879355911 +0000 UTC m=+549.711720886"
Jul 27 17:05:30.657851 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:30.657820    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8594521b-0c37-4657-a630-cd40ccb8de11" path="/var/lib/kubelet/pods/8594521b-0c37-4657-a630-cd40ccb8de11/volumes"
Jul 27 17:05:40.861499 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:40.861473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:46.079781 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.079752    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:46.080202 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.079937    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" podUID="9ca4694b-d9e7-4829-994e-9525db983307" containerName="manager" containerID="cri-o://30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9" gracePeriod=10
Jul 27 17:05:46.398210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.398184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:05:46.400501 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.400477    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.425673 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.425657    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:46.433644 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.433621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:05:46.481488 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481467    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume\") pod \"9ca4694b-d9e7-4829-994e-9525db983307\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") "
Jul 27 17:05:46.481585 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9f89\" (UniqueName: \"kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89\") pod \"9ca4694b-d9e7-4829-994e-9525db983307\" (UID: \"9ca4694b-d9e7-4829-994e-9525db983307\") "
Jul 27 17:05:46.481743 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmq9\" (UniqueName: \"kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.481850 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9ca4694b-d9e7-4829-994e-9525db983307" (UID: "9ca4694b-d9e7-4829-994e-9525db983307"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:05:46.481910 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.481955 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.481919    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca4694b-d9e7-4829-994e-9525db983307-extensions-socket-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:05:46.483536 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.483507    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89" (OuterVolumeSpecName: "kube-api-access-p9f89") pod "9ca4694b-d9e7-4829-994e-9525db983307" (UID: "9ca4694b-d9e7-4829-994e-9525db983307"). InnerVolumeSpecName "kube-api-access-p9f89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:05:46.582843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.582814    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.583036 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.582861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmq9\" (UniqueName: \"kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.583036 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.582924    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9f89\" (UniqueName: \"kubernetes.io/projected/9ca4694b-d9e7-4829-994e-9525db983307-kube-api-access-p9f89\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:05:46.583199 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.583175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.593474 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.593449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmq9\" (UniqueName: \"kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9\") pod \"kuadrant-operator-controller-manager-86878c769f-x9w4l\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.722546 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.722508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.871406 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:05:46.871376    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac28573_b335_4578_bd2e_ce3abd9d1dcd.slice/crio-788ef61e9028cfb66887ac299175833690c419d40fb43cdb861fb1aab75f09b2 WatchSource:0}: Error finding container 788ef61e9028cfb66887ac299175833690c419d40fb43cdb861fb1aab75f09b2: Status 404 returned error can't find the container with id 788ef61e9028cfb66887ac299175833690c419d40fb43cdb861fb1aab75f09b2
Jul 27 17:05:46.872402 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.872378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:05:46.925522 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.925493    2564 generic.go:356] "Generic (PLEG): container finished" podID="9ca4694b-d9e7-4829-994e-9525db983307" containerID="30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9" exitCode=0
Jul 27 17:05:46.925626 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.925550    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"
Jul 27 17:05:46.925626 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.925572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" event={"ID":"9ca4694b-d9e7-4829-994e-9525db983307","Type":"ContainerDied","Data":"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"}
Jul 27 17:05:46.925744 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.925662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw" event={"ID":"9ca4694b-d9e7-4829-994e-9525db983307","Type":"ContainerDied","Data":"9a9514b0ebc3c0858ff85037b679c8453c6dd302950ac8429993bfac485d8aa0"}
Jul 27 17:05:46.925744 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.925697    2564 scope.go:117] "RemoveContainer" containerID="30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"
Jul 27 17:05:46.927381 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.927326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" event={"ID":"aac28573-b335-4578-bd2e-ce3abd9d1dcd","Type":"ContainerStarted","Data":"788ef61e9028cfb66887ac299175833690c419d40fb43cdb861fb1aab75f09b2"}
Jul 27 17:05:46.927492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.927419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:05:46.933341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.933320    2564 scope.go:117] "RemoveContainer" containerID="30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"
Jul 27 17:05:46.933599 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:05:46.933571    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9\": container with ID starting with 30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9 not found: ID does not exist" containerID="30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"
Jul 27 17:05:46.933667 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.933601    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9"} err="failed to get container status \"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9\": rpc error: code = NotFound desc = could not find container \"30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9\": container with ID starting with 30904d55d6f1c40556eb39e2c21b1c40a5773a76455a7a6c43d3fd7f693582a9 not found: ID does not exist"
Jul 27 17:05:46.982920 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.982895    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:46.989743 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:46.989722    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h44nw"]
Jul 27 17:05:47.034007 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:47.033961    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" podStartSLOduration=1.033922274 podStartE2EDuration="1.033922274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:47.030709492 +0000 UTC m=+566.863074496" watchObservedRunningTime="2026-07-27 17:05:47.033922274 +0000 UTC m=+566.866287251"
Jul 27 17:05:47.933940 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:47.933908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" event={"ID":"aac28573-b335-4578-bd2e-ce3abd9d1dcd","Type":"ContainerStarted","Data":"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"}
Jul 27 17:05:48.658484 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:48.658443    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca4694b-d9e7-4829-994e-9525db983307" path="/var/lib/kubelet/pods/9ca4694b-d9e7-4829-994e-9525db983307/volumes"
Jul 27 17:05:58.940393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:05:58.940319    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:06:08.523557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.523523    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:06:08.524023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.523836    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca4694b-d9e7-4829-994e-9525db983307" containerName="manager"
Jul 27 17:06:08.524023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.523847    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca4694b-d9e7-4829-994e-9525db983307" containerName="manager"
Jul 27 17:06:08.524023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.523927    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca4694b-d9e7-4829-994e-9525db983307" containerName="manager"
Jul 27 17:06:08.526480 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.526461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.530093 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.530074    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qmxv5\""
Jul 27 17:06:08.530584 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.530568    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 17:06:08.541849 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.541827    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:06:08.587123 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.587104    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:06:08.649686 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.649661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21e31e58-e858-420f-beba-75d6d9d0c79d-config-file\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.649773 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.649712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtx6k\" (UniqueName: \"kubernetes.io/projected/21e31e58-e858-420f-beba-75d6d9d0c79d-kube-api-access-xtx6k\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.750831 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.750806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx6k\" (UniqueName: \"kubernetes.io/projected/21e31e58-e858-420f-beba-75d6d9d0c79d-kube-api-access-xtx6k\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.750963 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.750848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21e31e58-e858-420f-beba-75d6d9d0c79d-config-file\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.751395 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.751378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21e31e58-e858-420f-beba-75d6d9d0c79d-config-file\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.761439 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.761421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx6k\" (UniqueName: \"kubernetes.io/projected/21e31e58-e858-420f-beba-75d6d9d0c79d-kube-api-access-xtx6k\") pod \"limitador-limitador-54d6d988d5-jx88v\" (UID: \"21e31e58-e858-420f-beba-75d6d9d0c79d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.837048 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.836983    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:08.962499 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:08.962473    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:06:08.966163 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:06:08.966131    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e31e58_e858_420f_beba_75d6d9d0c79d.slice/crio-41dd54ff00c702345e5fcae025cf8ed93d23ef1ec41c5a88a6cc0cc180533217 WatchSource:0}: Error finding container 41dd54ff00c702345e5fcae025cf8ed93d23ef1ec41c5a88a6cc0cc180533217: Status 404 returned error can't find the container with id 41dd54ff00c702345e5fcae025cf8ed93d23ef1ec41c5a88a6cc0cc180533217
Jul 27 17:06:09.024189 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:09.024161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v" event={"ID":"21e31e58-e858-420f-beba-75d6d9d0c79d","Type":"ContainerStarted","Data":"41dd54ff00c702345e5fcae025cf8ed93d23ef1ec41c5a88a6cc0cc180533217"}
Jul 27 17:06:13.044282 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:13.044246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v" event={"ID":"21e31e58-e858-420f-beba-75d6d9d0c79d","Type":"ContainerStarted","Data":"c676a5128f777dcbfd66adc53407a65d62a47ed950b3c022d9d1d7b3d0321dc3"}
Jul 27 17:06:13.044668 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:13.044306    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:13.064711 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:13.064666    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v" podStartSLOduration=1.894482688 podStartE2EDuration="5.064651798s" totalImagesPullingTime="3.17016911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:13.063353317 +0000 UTC m=+592.895718337" watchObservedRunningTime="2026-07-27 17:06:13.064651798 +0000 UTC m=+592.897016774"
Jul 27 17:06:20.576557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:20.576529    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:06:20.577869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:20.577847    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:06:24.048988 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:24.048956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jx88v"
Jul 27 17:06:46.299523 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.299449    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:46.302872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.302855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:46.318137 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.318116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:46.318747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.318734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t2r6d\""
Jul 27 17:06:46.339688 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.339666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzsgc\" (UniqueName: \"kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc\") pod \"authorino-5f868c4b4f-khgtw\" (UID: \"ad704f50-cd04-4834-8e66-1f25fba978b6\") " pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:46.440152 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.440126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzsgc\" (UniqueName: \"kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc\") pod \"authorino-5f868c4b4f-khgtw\" (UID: \"ad704f50-cd04-4834-8e66-1f25fba978b6\") " pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:46.460551 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.460530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzsgc\" (UniqueName: \"kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc\") pod \"authorino-5f868c4b4f-khgtw\" (UID: \"ad704f50-cd04-4834-8e66-1f25fba978b6\") " pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:46.611335 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.611280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:46.658802 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.658776    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:46.743344 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.743318    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:06:46.747949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.747932    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.751850 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.751831    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 17:06:46.781959 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.781934    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:06:46.804367 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.804349    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:46.806816 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:06:46.806794    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad704f50_cd04_4834_8e66_1f25fba978b6.slice/crio-e45e3cab017f90d36cbbceed7f4fd98f00f87123a1102c989bce0645f03c9368 WatchSource:0}: Error finding container e45e3cab017f90d36cbbceed7f4fd98f00f87123a1102c989bce0645f03c9368: Status 404 returned error can't find the container with id e45e3cab017f90d36cbbceed7f4fd98f00f87123a1102c989bce0645f03c9368
Jul 27 17:06:46.844184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.844162    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g57r7\" (UniqueName: \"kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.844279 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.844217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.945186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.945159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.945281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.945206    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g57r7\" (UniqueName: \"kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.947543 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.947521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:46.961971 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:46.961950    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g57r7\" (UniqueName: \"kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7\") pod \"authorino-5b6d7565b9-ln8hs\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") " pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:47.057679 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:47.057657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:06:47.195046 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:47.195013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" event={"ID":"ad704f50-cd04-4834-8e66-1f25fba978b6","Type":"ContainerStarted","Data":"e45e3cab017f90d36cbbceed7f4fd98f00f87123a1102c989bce0645f03c9368"}
Jul 27 17:06:47.230486 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:47.229835    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:06:47.234300 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:06:47.234273    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4839a206_58c8_40d6_a3ed_879740e31369.slice/crio-da93e5539f676e669eb2e8d7a96701c34bfe903d53c1b256d284f81d00e43167 WatchSource:0}: Error finding container da93e5539f676e669eb2e8d7a96701c34bfe903d53c1b256d284f81d00e43167: Status 404 returned error can't find the container with id da93e5539f676e669eb2e8d7a96701c34bfe903d53c1b256d284f81d00e43167
Jul 27 17:06:48.204865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:48.204833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" event={"ID":"4839a206-58c8-40d6-a3ed-879740e31369","Type":"ContainerStarted","Data":"da93e5539f676e669eb2e8d7a96701c34bfe903d53c1b256d284f81d00e43167"}
Jul 27 17:06:50.216540 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.216498    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" event={"ID":"ad704f50-cd04-4834-8e66-1f25fba978b6","Type":"ContainerStarted","Data":"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"}
Jul 27 17:06:50.217026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.216552    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" podUID="ad704f50-cd04-4834-8e66-1f25fba978b6" containerName="authorino" containerID="cri-o://05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a" gracePeriod=30
Jul 27 17:06:50.218469 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.218436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" event={"ID":"4839a206-58c8-40d6-a3ed-879740e31369","Type":"ContainerStarted","Data":"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"}
Jul 27 17:06:50.281913 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.281814    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" podStartSLOduration=1.072011599 podStartE2EDuration="4.28180206s" totalImagesPullingTime="3.209790461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:50.242766999 +0000 UTC m=+630.075131986" watchObservedRunningTime="2026-07-27 17:06:50.28180206 +0000 UTC m=+630.114167035"
Jul 27 17:06:50.449391 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.449364    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:50.480955 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.480916    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" podStartSLOduration=1.7098261620000001 podStartE2EDuration="4.480900429s" totalImagesPullingTime="2.771074267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:50.283392617 +0000 UTC m=+630.115757592" watchObservedRunningTime="2026-07-27 17:06:50.480900429 +0000 UTC m=+630.313265406"
Jul 27 17:06:50.575433 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.575376    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzsgc\" (UniqueName: \"kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc\") pod \"ad704f50-cd04-4834-8e66-1f25fba978b6\" (UID: \"ad704f50-cd04-4834-8e66-1f25fba978b6\") "
Jul 27 17:06:50.577292 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.577272    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc" (OuterVolumeSpecName: "kube-api-access-dzsgc") pod "ad704f50-cd04-4834-8e66-1f25fba978b6" (UID: "ad704f50-cd04-4834-8e66-1f25fba978b6"). InnerVolumeSpecName "kube-api-access-dzsgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:06:50.676223 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:50.676205    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzsgc\" (UniqueName: \"kubernetes.io/projected/ad704f50-cd04-4834-8e66-1f25fba978b6-kube-api-access-dzsgc\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:06:51.223713 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.223678    2564 generic.go:356] "Generic (PLEG): container finished" podID="ad704f50-cd04-4834-8e66-1f25fba978b6" containerID="05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a" exitCode=0
Jul 27 17:06:51.224246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.223726    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f868c4b4f-khgtw"
Jul 27 17:06:51.224246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.223767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" event={"ID":"ad704f50-cd04-4834-8e66-1f25fba978b6","Type":"ContainerDied","Data":"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"}
Jul 27 17:06:51.224246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.223811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f868c4b4f-khgtw" event={"ID":"ad704f50-cd04-4834-8e66-1f25fba978b6","Type":"ContainerDied","Data":"e45e3cab017f90d36cbbceed7f4fd98f00f87123a1102c989bce0645f03c9368"}
Jul 27 17:06:51.224246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.223829    2564 scope.go:117] "RemoveContainer" containerID="05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"
Jul 27 17:06:51.231839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.231819    2564 scope.go:117] "RemoveContainer" containerID="05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"
Jul 27 17:06:51.232114 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:06:51.232094    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a\": container with ID starting with 05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a not found: ID does not exist" containerID="05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"
Jul 27 17:06:51.232176 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.232122    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a"} err="failed to get container status \"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a\": rpc error: code = NotFound desc = could not find container \"05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a\": container with ID starting with 05cdcf6c7309856d36a90cc61810de64c27546017a3e8ce0ba825733498d792a not found: ID does not exist"
Jul 27 17:06:51.295765 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.295721    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:51.307125 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:51.307106    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f868c4b4f-khgtw"]
Jul 27 17:06:52.657383 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:06:52.657343    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad704f50-cd04-4834-8e66-1f25fba978b6" path="/var/lib/kubelet/pods/ad704f50-cd04-4834-8e66-1f25fba978b6/volumes"
Jul 27 17:07:33.368705 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:07:33.368669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:08:11.729506 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:08:11.729469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:08:27.605105 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:08:27.605068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:08:41.397678 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:08:41.397642    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:09:01.788801 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:01.788759    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:09:11.691002 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:11.690970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:09:15.823090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.823058    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"]
Jul 27 17:09:15.823459 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.823387    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad704f50-cd04-4834-8e66-1f25fba978b6" containerName="authorino"
Jul 27 17:09:15.823459 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.823398    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad704f50-cd04-4834-8e66-1f25fba978b6" containerName="authorino"
Jul 27 17:09:15.823459 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.823447    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad704f50-cd04-4834-8e66-1f25fba978b6" containerName="authorino"
Jul 27 17:09:15.825523 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.825503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.829567 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.829547    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 17:09:15.830923 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.830891    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6x9nq\""
Jul 27 17:09:15.831028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.831003    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 17:09:15.831028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.831023    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 17:09:15.840193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.840173    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"]
Jul 27 17:09:15.893773 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.893937 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.893937 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.893937 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trg2w\" (UniqueName: \"kubernetes.io/projected/3eda460f-7cc5-46c4-9117-4f85e98a1df4-kube-api-access-trg2w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.894050 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.894050 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.893964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995227 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995194    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995227 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995499 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995499 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995307    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995499 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trg2w\" (UniqueName: \"kubernetes.io/projected/3eda460f-7cc5-46c4-9117-4f85e98a1df4-kube-api-access-trg2w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995716 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995662    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995756 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.995851 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.995837    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.998186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.998162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eda460f-7cc5-46c4-9117-4f85e98a1df4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:15.998355 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:15.998332    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eda460f-7cc5-46c4-9117-4f85e98a1df4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:16.018950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:16.018915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trg2w\" (UniqueName: \"kubernetes.io/projected/3eda460f-7cc5-46c4-9117-4f85e98a1df4-kube-api-access-trg2w\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-hcczl\" (UID: \"3eda460f-7cc5-46c4-9117-4f85e98a1df4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:16.134908 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:16.134804    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:16.305888 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:16.305849    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"]
Jul 27 17:09:16.306017 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:09:16.305994    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eda460f_7cc5_46c4_9117_4f85e98a1df4.slice/crio-a7edae5ac86988ae51a462cc2acde2de91d9a7020666a2c8b1e3687ab2f1993d WatchSource:0}: Error finding container a7edae5ac86988ae51a462cc2acde2de91d9a7020666a2c8b1e3687ab2f1993d: Status 404 returned error can't find the container with id a7edae5ac86988ae51a462cc2acde2de91d9a7020666a2c8b1e3687ab2f1993d
Jul 27 17:09:16.307328 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:16.307312    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:09:16.871798 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:16.871761    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl" event={"ID":"3eda460f-7cc5-46c4-9117-4f85e98a1df4","Type":"ContainerStarted","Data":"a7edae5ac86988ae51a462cc2acde2de91d9a7020666a2c8b1e3687ab2f1993d"}
Jul 27 17:09:17.878454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:17.878373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl" event={"ID":"3eda460f-7cc5-46c4-9117-4f85e98a1df4","Type":"ContainerStarted","Data":"bb6541fb195e9c650b7a33cc2b1b240a4698b6894ed9345308fade6e45a2b32a"}
Jul 27 17:09:17.904624 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:17.904563    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl" podStartSLOduration=1.75417597 podStartE2EDuration="2.904543102s" totalImagesPullingTime="1.150367132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:17.901812063 +0000 UTC m=+777.734177039" watchObservedRunningTime="2026-07-27 17:09:17.904543102 +0000 UTC m=+777.736908081"
Jul 27 17:09:20.496864 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:20.496822    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:09:26.135247 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:26.135215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:26.135684 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:26.135455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:26.147447 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:26.147421    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:09:26.927670 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:09:26.927633    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-hcczl"
Jul 27 17:10:58.860844 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:58.860805    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:10:58.861445 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:58.861007    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" podUID="4839a206-58c8-40d6-a3ed-879740e31369" containerName="authorino" containerID="cri-o://dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b" gracePeriod=30
Jul 27 17:10:59.108080 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.108058    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:10:59.199735 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.199707    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g57r7\" (UniqueName: \"kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7\") pod \"4839a206-58c8-40d6-a3ed-879740e31369\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") "
Jul 27 17:10:59.199865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.199737    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert\") pod \"4839a206-58c8-40d6-a3ed-879740e31369\" (UID: \"4839a206-58c8-40d6-a3ed-879740e31369\") "
Jul 27 17:10:59.201753 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.201727    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7" (OuterVolumeSpecName: "kube-api-access-g57r7") pod "4839a206-58c8-40d6-a3ed-879740e31369" (UID: "4839a206-58c8-40d6-a3ed-879740e31369"). InnerVolumeSpecName "kube-api-access-g57r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:59.209023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.208998    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4839a206-58c8-40d6-a3ed-879740e31369" (UID: "4839a206-58c8-40d6-a3ed-879740e31369"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:10:59.300445 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.300419    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g57r7\" (UniqueName: \"kubernetes.io/projected/4839a206-58c8-40d6-a3ed-879740e31369-kube-api-access-g57r7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:10:59.300545 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.300446    2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4839a206-58c8-40d6-a3ed-879740e31369-tls-cert\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:10:59.308038 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.308011    2564 generic.go:356] "Generic (PLEG): container finished" podID="4839a206-58c8-40d6-a3ed-879740e31369" containerID="dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b" exitCode=0
Jul 27 17:10:59.308135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.308056    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs"
Jul 27 17:10:59.308135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.308067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" event={"ID":"4839a206-58c8-40d6-a3ed-879740e31369","Type":"ContainerDied","Data":"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"}
Jul 27 17:10:59.308135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.308105    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b6d7565b9-ln8hs" event={"ID":"4839a206-58c8-40d6-a3ed-879740e31369","Type":"ContainerDied","Data":"da93e5539f676e669eb2e8d7a96701c34bfe903d53c1b256d284f81d00e43167"}
Jul 27 17:10:59.308135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.308127    2564 scope.go:117] "RemoveContainer" containerID="dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"
Jul 27 17:10:59.316220 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.316206    2564 scope.go:117] "RemoveContainer" containerID="dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"
Jul 27 17:10:59.316450 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:10:59.316433    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b\": container with ID starting with dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b not found: ID does not exist" containerID="dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"
Jul 27 17:10:59.316510 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.316462    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b"} err="failed to get container status \"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b\": rpc error: code = NotFound desc = could not find container \"dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b\": container with ID starting with dbb5feeb47733cb9ab4923eee2d4e80f38d03dd9773478e8ac0bc63badf4e44b not found: ID does not exist"
Jul 27 17:10:59.334393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.334372    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:10:59.343073 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:10:59.343055    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b6d7565b9-ln8hs"]
Jul 27 17:11:00.657789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:00.657760    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4839a206-58c8-40d6-a3ed-879740e31369" path="/var/lib/kubelet/pods/4839a206-58c8-40d6-a3ed-879740e31369/volumes"
Jul 27 17:11:20.605172 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:20.605144    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:11:20.605860 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:20.605842    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:11:26.294001 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:26.293965    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:11:37.198093 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:37.198060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:11:47.520681 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:11:47.520650    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:13:32.320120 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:13:32.320085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:13:41.706863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:13:41.706832    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:14:11.894752 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:14:11.894721    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:14:21.889634 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:14:21.889602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:14:30.793962 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:14:30.793928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:14:39.639417 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:14:39.639385    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:14:50.504055 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:14:50.504022    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:00.105791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:00.105757    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:11.005262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:11.005230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:20.789309 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:20.789276    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:29.500696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:29.500663    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:36.917860 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:36.917825    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:15:53.004378 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:15:53.004304    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:16:05.495490 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:05.495458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:16:20.635704 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:20.635676    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:16:20.637740 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:20.637718    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:16:29.990623 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:29.990591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:16:40.285360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:40.285326    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:16:56.000258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:16:56.000224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:17:04.823014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:04.822981    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:17:14.394912 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:14.394873    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:17:23.379705 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:23.379673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:17:33.696089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:33.696055    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:17:59.850099 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:59.850065    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:17:59.850578 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:17:59.850309    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" podUID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" containerName="manager" containerID="cri-o://b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad" gracePeriod=10
Jul 27 17:18:02.896344 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:02.896321    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:18:02.956745 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:02.956723    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cmq9\" (UniqueName: \"kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9\") pod \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") "
Jul 27 17:18:02.956874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:02.956826    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume\") pod \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\" (UID: \"aac28573-b335-4578-bd2e-ce3abd9d1dcd\") "
Jul 27 17:18:02.957262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:02.957233    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aac28573-b335-4578-bd2e-ce3abd9d1dcd" (UID: "aac28573-b335-4578-bd2e-ce3abd9d1dcd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:18:02.958663 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:02.958639    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9" (OuterVolumeSpecName: "kube-api-access-8cmq9") pod "aac28573-b335-4578-bd2e-ce3abd9d1dcd" (UID: "aac28573-b335-4578-bd2e-ce3abd9d1dcd"). InnerVolumeSpecName "kube-api-access-8cmq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:18:03.057505 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.057456    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cmq9\" (UniqueName: \"kubernetes.io/projected/aac28573-b335-4578-bd2e-ce3abd9d1dcd-kube-api-access-8cmq9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:18:03.057505 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.057475    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac28573-b335-4578-bd2e-ce3abd9d1dcd-extensions-socket-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:18:03.128437 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.128413    2564 generic.go:356] "Generic (PLEG): container finished" podID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" containerID="b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad" exitCode=0
Jul 27 17:18:03.128526 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.128460    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"
Jul 27 17:18:03.128526 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.128491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" event={"ID":"aac28573-b335-4578-bd2e-ce3abd9d1dcd","Type":"ContainerDied","Data":"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"}
Jul 27 17:18:03.128598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.128535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l" event={"ID":"aac28573-b335-4578-bd2e-ce3abd9d1dcd","Type":"ContainerDied","Data":"788ef61e9028cfb66887ac299175833690c419d40fb43cdb861fb1aab75f09b2"}
Jul 27 17:18:03.128598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.128562    2564 scope.go:117] "RemoveContainer" containerID="b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"
Jul 27 17:18:03.137784 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.137765    2564 scope.go:117] "RemoveContainer" containerID="b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"
Jul 27 17:18:03.138058 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:18:03.138039    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad\": container with ID starting with b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad not found: ID does not exist" containerID="b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"
Jul 27 17:18:03.138123 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.138065    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad"} err="failed to get container status \"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad\": rpc error: code = NotFound desc = could not find container \"b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad\": container with ID starting with b510cff9794b11c6c3468b9999881510193adf54554e0d95e344962a99c389ad not found: ID does not exist"
Jul 27 17:18:03.151275 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.151257    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:18:03.155844 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:03.155826    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x9w4l"]
Jul 27 17:18:04.657737 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:18:04.657704    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" path="/var/lib/kubelet/pods/aac28573-b335-4578-bd2e-ce3abd9d1dcd/volumes"
Jul 27 17:19:00.969796 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.969721    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"]
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970075    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4839a206-58c8-40d6-a3ed-879740e31369" containerName="authorino"
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970087    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4839a206-58c8-40d6-a3ed-879740e31369" containerName="authorino"
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970105    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" containerName="manager"
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970110    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" containerName="manager"
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970172    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac28573-b335-4578-bd2e-ce3abd9d1dcd" containerName="manager"
Jul 27 17:19:00.970245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.970182    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4839a206-58c8-40d6-a3ed-879740e31369" containerName="authorino"
Jul 27 17:19:00.972049 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.972032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:00.975068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.975045    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rs4tp\""
Jul 27 17:19:00.990937 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.990911    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"]
Jul 27 17:19:00.997238 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.997216    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllfg\" (UniqueName: \"kubernetes.io/projected/81dcacb1-9a9c-468d-be3b-a89d11b50a37-kube-api-access-bllfg\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:00.997360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:00.997247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dcacb1-9a9c-468d-be3b-a89d11b50a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.098139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.098112    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllfg\" (UniqueName: \"kubernetes.io/projected/81dcacb1-9a9c-468d-be3b-a89d11b50a37-kube-api-access-bllfg\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.098271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.098154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dcacb1-9a9c-468d-be3b-a89d11b50a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.098528 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.098496    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/81dcacb1-9a9c-468d-be3b-a89d11b50a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.110709 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.110685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllfg\" (UniqueName: \"kubernetes.io/projected/81dcacb1-9a9c-468d-be3b-a89d11b50a37-kube-api-access-bllfg\") pod \"kuadrant-operator-controller-manager-86878c769f-wzd6l\" (UID: \"81dcacb1-9a9c-468d-be3b-a89d11b50a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.281610 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.281552    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:01.415073 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:01.415041    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"]
Jul 27 17:19:01.416113 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:19:01.416090    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dcacb1_9a9c_468d_be3b_a89d11b50a37.slice/crio-5a43db1ae20951c34224bd4a1f3c0c628632247a978b0b673e4bd32dda7502b6 WatchSource:0}: Error finding container 5a43db1ae20951c34224bd4a1f3c0c628632247a978b0b673e4bd32dda7502b6: Status 404 returned error can't find the container with id 5a43db1ae20951c34224bd4a1f3c0c628632247a978b0b673e4bd32dda7502b6
Jul 27 17:19:02.394062 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:02.394027    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l" event={"ID":"81dcacb1-9a9c-468d-be3b-a89d11b50a37","Type":"ContainerStarted","Data":"f8290781e58676ee5e1581b376057e31a3f7cdfd683c4d5b557c345fa2d960ef"}
Jul 27 17:19:02.394457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:02.394083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l" event={"ID":"81dcacb1-9a9c-468d-be3b-a89d11b50a37","Type":"ContainerStarted","Data":"5a43db1ae20951c34224bd4a1f3c0c628632247a978b0b673e4bd32dda7502b6"}
Jul 27 17:19:02.394457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:02.394106    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:02.425551 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:02.425502    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l" podStartSLOduration=2.425487174 podStartE2EDuration="2.425487174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:19:02.425369498 +0000 UTC m=+1362.257734498" watchObservedRunningTime="2026-07-27 17:19:02.425487174 +0000 UTC m=+1362.257852151"
Jul 27 17:19:13.400001 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:13.399969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzd6l"
Jul 27 17:19:24.021956 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:24.021923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:19:32.827431 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:32.827397    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:19:49.498320 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:19:49.498286    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:20:10.025576 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:20:10.025536    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:20:21.623276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:20:21.623242    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:20:32.022897 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:20:32.022852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:20:43.006001 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:20:43.005963    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:20:51.008080 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:20:51.008051    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:21:01.787090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:01.787052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:21:13.327893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:13.327837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:21:20.667657 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:20.667628    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:21:20.669999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:20.669980    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:21:20.896533 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:20.896495    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:21:41.596946 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:41.596908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:21:49.192696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:21:49.192622    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:22:00.715893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.715847    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:22:00.719178 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.719158    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.722161 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.722129    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-31437c34-kserve-self-signed-certs\""
Jul 27 17:22:00.739759 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.739737    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:22:00.884067 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.884210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhgb\" (UniqueName: \"kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.884210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.884210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.884335 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.884335 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.884247    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984580    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhgb\" (UniqueName: \"kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.984863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.985118 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.984945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.985260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.985240    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.985353 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.985327    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.987147 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.987127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.987485 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.987466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:00.994414 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:00.994392    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhgb\" (UniqueName: \"kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb\") pod \"e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") " pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:01.027861 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:01.027838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:01.160547 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:01.160522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:22:01.162524 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:22:01.162494    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c83fbe2_ae46_4b0d_8f01_ac63ac4161a2.slice/crio-b793b71d98da854aed658d6844e0d9f453b2b6cfca3326062d52c52291b8c7e1 WatchSource:0}: Error finding container b793b71d98da854aed658d6844e0d9f453b2b6cfca3326062d52c52291b8c7e1: Status 404 returned error can't find the container with id b793b71d98da854aed658d6844e0d9f453b2b6cfca3326062d52c52291b8c7e1
Jul 27 17:22:01.180375 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:01.180347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" event={"ID":"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2","Type":"ContainerStarted","Data":"b793b71d98da854aed658d6844e0d9f453b2b6cfca3326062d52c52291b8c7e1"}
Jul 27 17:22:02.186671 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:02.186638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" event={"ID":"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2","Type":"ContainerStarted","Data":"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"}
Jul 27 17:22:02.210098 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:02.210054    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" podStartSLOduration=2.21003984 podStartE2EDuration="2.21003984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:22:02.207696468 +0000 UTC m=+1542.040061444" watchObservedRunningTime="2026-07-27 17:22:02.21003984 +0000 UTC m=+1542.042404816"
Jul 27 17:22:03.710843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:03.710811    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:22:11.028495 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:11.028411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:11.028495 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:11.028467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:11.041083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:11.041059    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:11.240265 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:11.240233    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:22:22.421699 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.421667    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:22:22.424783 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.424765    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.427790 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.427768    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-31437c34-kserve-self-signed-certs\""
Jul 27 17:22:22.441236 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.441214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:22:22.456492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.456597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.456597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.456692 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456597    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.456692 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456615    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fktwz\" (UniqueName: \"kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.456692 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.456669    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557606 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557581    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.557930 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.557821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fktwz\" (UniqueName: \"kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.558112 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.558068    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.558250 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.558235    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.558307 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.558281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.559945 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.559930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.560171 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.560151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.567843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.567811    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fktwz\" (UniqueName: \"kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz\") pod \"e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") " pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.734387 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.734322    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:22.862341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:22.862313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:22:22.865726 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:22:22.865701    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f57b90a_0ab2_43a7_ab02_ffc9337ae673.slice/crio-22ac8636311cc4cdc1fa3d6ffed0458d3143e2c335d732d7959bdeec6feb1b25 WatchSource:0}: Error finding container 22ac8636311cc4cdc1fa3d6ffed0458d3143e2c335d732d7959bdeec6feb1b25: Status 404 returned error can't find the container with id 22ac8636311cc4cdc1fa3d6ffed0458d3143e2c335d732d7959bdeec6feb1b25
Jul 27 17:22:23.282790 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:23.282754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" event={"ID":"2f57b90a-0ab2-43a7-ab02-ffc9337ae673","Type":"ContainerStarted","Data":"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"}
Jul 27 17:22:23.283010 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:23.282798    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" event={"ID":"2f57b90a-0ab2-43a7-ab02-ffc9337ae673","Type":"ContainerStarted","Data":"22ac8636311cc4cdc1fa3d6ffed0458d3143e2c335d732d7959bdeec6feb1b25"}
Jul 27 17:22:23.304151 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:23.304106    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" podStartSLOduration=1.304091189 podStartE2EDuration="1.304091189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:22:23.304067946 +0000 UTC m=+1563.136432922" watchObservedRunningTime="2026-07-27 17:22:23.304091189 +0000 UTC m=+1563.136456143"
Jul 27 17:22:25.943630 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:25.943587    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:22:32.735436 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:32.735401    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:32.735849 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:32.735450    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:32.748380 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:32.748355    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:33.337786 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:33.337760    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:22:59.736218 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:22:59.736181    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:23:01.433647 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:01.433612    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:23:02.559755 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:02.559726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:23:14.754251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:14.754212    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" podUID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" containerName="main" containerID="cri-o://a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf" gracePeriod=45
Jul 27 17:23:14.989121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:14.989096    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:23:15.070805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070736    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnhgb\" (UniqueName: \"kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.070805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.071047 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070810    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.071047 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070846    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.071047 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070868    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.071047 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.070908    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir\") pod \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\" (UID: \"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2\") "
Jul 27 17:23:15.071047 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.071007    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:15.071288 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.071179    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:15.071288 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.071194    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.071288 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.071250    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home" (OuterVolumeSpecName: "home") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:15.073151 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.073126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm" (OuterVolumeSpecName: "dshm") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:15.073243 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.073181    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb" (OuterVolumeSpecName: "kube-api-access-hnhgb") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "kube-api-access-hnhgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:23:15.073243 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.073181    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" (UID: "7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:23:15.172315 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.172288    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnhgb\" (UniqueName: \"kubernetes.io/projected/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-kube-api-access-hnhgb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.172315 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.172312    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.172462 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.172323    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.172462 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.172331    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.172462 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.172340    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:15.515323 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.515288    2564 generic.go:356] "Generic (PLEG): container finished" podID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" containerID="a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf" exitCode=0
Jul 27 17:23:15.515481 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.515334    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"
Jul 27 17:23:15.515481 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.515379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" event={"ID":"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2","Type":"ContainerDied","Data":"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"}
Jul 27 17:23:15.515481 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.515431    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2" event={"ID":"7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2","Type":"ContainerDied","Data":"b793b71d98da854aed658d6844e0d9f453b2b6cfca3326062d52c52291b8c7e1"}
Jul 27 17:23:15.515481 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.515455    2564 scope.go:117] "RemoveContainer" containerID="a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"
Jul 27 17:23:15.524536 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.524517    2564 scope.go:117] "RemoveContainer" containerID="a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"
Jul 27 17:23:15.524799 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:23:15.524779    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf\": container with ID starting with a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf not found: ID does not exist" containerID="a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"
Jul 27 17:23:15.524893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.524807    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf"} err="failed to get container status \"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf\": rpc error: code = NotFound desc = could not find container \"a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf\": container with ID starting with a3ad375bb77198d0aedf0bb841396ad3f893cbb50754c0afbf4110426d571fdf not found: ID does not exist"
Jul 27 17:23:15.538717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.538694    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:23:15.542308 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.542290    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-31437c34-kserve-5985b54c95-ln9w2"]
Jul 27 17:23:15.593914 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:15.593891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:23:16.452046 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.452001    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" podUID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" containerName="main" containerID="cri-o://b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187" gracePeriod=45
Jul 27 17:23:16.659238 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.659199    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" path="/var/lib/kubelet/pods/7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2/volumes"
Jul 27 17:23:16.694304 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.694284    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:23:16.784781 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784717    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.784781 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784768    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fktwz\" (UniqueName: \"kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.784982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784800    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.784982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.784982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784860    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.784982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.784918    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache\") pod \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\" (UID: \"2f57b90a-0ab2-43a7-ab02-ffc9337ae673\") "
Jul 27 17:23:16.785181 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.785022    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home" (OuterVolumeSpecName: "home") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:16.785181 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.785097    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:16.785295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.785209    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:16.785295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.785222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:16.785295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.785233    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:16.786862 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.786840    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm" (OuterVolumeSpecName: "dshm") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:16.786985 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.786908    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz" (OuterVolumeSpecName: "kube-api-access-fktwz") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "kube-api-access-fktwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:23:16.786985 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.786928    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f57b90a-0ab2-43a7-ab02-ffc9337ae673" (UID: "2f57b90a-0ab2-43a7-ab02-ffc9337ae673"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:23:16.885968 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.885937    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:16.885968 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.885957    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:16.885968 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.885969    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fktwz\" (UniqueName: \"kubernetes.io/projected/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-kube-api-access-fktwz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:16.886158 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:16.885978    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f57b90a-0ab2-43a7-ab02-ffc9337ae673-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:23:17.532621 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.532583    2564 generic.go:356] "Generic (PLEG): container finished" podID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" containerID="b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187" exitCode=0
Jul 27 17:23:17.533101 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.532629    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"
Jul 27 17:23:17.533101 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.532669    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" event={"ID":"2f57b90a-0ab2-43a7-ab02-ffc9337ae673","Type":"ContainerDied","Data":"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"}
Jul 27 17:23:17.533101 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.532720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6" event={"ID":"2f57b90a-0ab2-43a7-ab02-ffc9337ae673","Type":"ContainerDied","Data":"22ac8636311cc4cdc1fa3d6ffed0458d3143e2c335d732d7959bdeec6feb1b25"}
Jul 27 17:23:17.533101 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.532737    2564 scope.go:117] "RemoveContainer" containerID="b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"
Jul 27 17:23:17.541558 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.541543    2564 scope.go:117] "RemoveContainer" containerID="b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"
Jul 27 17:23:17.541806 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:23:17.541787    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187\": container with ID starting with b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187 not found: ID does not exist" containerID="b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"
Jul 27 17:23:17.541855 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.541812    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187"} err="failed to get container status \"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187\": rpc error: code = NotFound desc = could not find container \"b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187\": container with ID starting with b103f4e35c79d698fa5b06f19960c4c5e09ba4c9a6278ce44b874fc3d362e187 not found: ID does not exist"
Jul 27 17:23:17.570154 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.570134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:23:17.576200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:17.576173    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-31437c34-kserve-6d6dff9875-x6lv6"]
Jul 27 17:23:18.657493 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:18.657460    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" path="/var/lib/kubelet/pods/2f57b90a-0ab2-43a7-ab02-ffc9337ae673/volumes"
Jul 27 17:23:24.496869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:24.496836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:23:37.394102 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:37.394069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:23:46.301470 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:46.301437    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:23:55.992788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:23:55.992754    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:03.712083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:03.712047    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:14.292655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:14.292619    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:23.787940 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:23.787904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:36.790870 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:36.790842    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:45.688918 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:45.688818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:24:55.411251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:24:55.411215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:25:07.892670 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:25:07.892639    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:25:17.896913 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:25:17.896868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:25:27.300934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:25:27.300901    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:25:38.496678 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:25:38.496645    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:25:49.915756 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:25:49.915726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:00.090356 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:00.090324    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:13.840189 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:13.840111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:20.698537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:20.698511    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:26:20.701266 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:20.701240    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:26:23.685982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:23.685951    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:32.905457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:32.905418    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:37.146747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.146715    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147068    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" containerName="main"
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147081    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" containerName="main"
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147099    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" containerName="main"
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147105    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" containerName="main"
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147153    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c83fbe2-ae46-4b0d-8f01-ac63ac4161a2" containerName="main"
Jul 27 17:26:37.147251 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.147161    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f57b90a-0ab2-43a7-ab02-ffc9337ae673" containerName="main"
Jul 27 17:26:37.150142 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.150120    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.155354 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.155332    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:26:37.155478 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.155357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-41d9ef37-serving-cert\""
Jul 27 17:26:37.156052 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.156032    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:26:37.156052 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.156044    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:26:37.164369 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.164352    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:26:37.250914 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.250854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.251089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.250953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qg6\" (UniqueName: \"kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.251089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.250982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.351797 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.351771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.351973 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.351812    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qg6\" (UniqueName: \"kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.351973 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.351829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.352423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.352404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.354362 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.354338    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.364017 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.363993    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qg6\" (UniqueName: \"kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6\") pod \"maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.460112 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.460058    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:37.593722 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.593694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:26:37.597161 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:26:37.597134    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55775264_6e02_4184_ab8b_6cb8c29f0ea2.slice/crio-c097c99be33a58777251b543207841d60f9747056c595419133171cb43348843 WatchSource:0}: Error finding container c097c99be33a58777251b543207841d60f9747056c595419133171cb43348843: Status 404 returned error can't find the container with id c097c99be33a58777251b543207841d60f9747056c595419133171cb43348843
Jul 27 17:26:37.598861 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:37.598846    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:26:38.391073 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:38.391039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" event={"ID":"55775264-6e02-4184-ab8b-6cb8c29f0ea2","Type":"ContainerStarted","Data":"c097c99be33a58777251b543207841d60f9747056c595419133171cb43348843"}
Jul 27 17:26:40.402187 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:40.402154    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" event={"ID":"55775264-6e02-4184-ab8b-6cb8c29f0ea2","Type":"ContainerStarted","Data":"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"}
Jul 27 17:26:40.402649 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:40.402247    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:26:40.425564 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:40.425518    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" podStartSLOduration=1.12895259 podStartE2EDuration="3.425503793s" totalImagesPullingTime="2.296551203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:26:40.424734596 +0000 UTC m=+1820.257099583" watchObservedRunningTime="2026-07-27 17:26:40.425503793 +0000 UTC m=+1820.257868769"
Jul 27 17:26:44.091817 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:44.091780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:26:46.411811 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:26:46.411777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:27:13.118695 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.118661    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:27:13.119224 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.118944    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" podUID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" containerName="maas-api" containerID="cri-o://2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374" gracePeriod=30
Jul 27 17:27:13.369169 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.369107    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:27:13.429985 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.429951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls\") pod \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") "
Jul 27 17:27:13.430170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.429996    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca\") pod \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") "
Jul 27 17:27:13.430170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.430065    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2qg6\" (UniqueName: \"kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6\") pod \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\" (UID: \"55775264-6e02-4184-ab8b-6cb8c29f0ea2\") "
Jul 27 17:27:13.430443 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.430418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55775264-6e02-4184-ab8b-6cb8c29f0ea2" (UID: "55775264-6e02-4184-ab8b-6cb8c29f0ea2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:27:13.432388 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.432363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55775264-6e02-4184-ab8b-6cb8c29f0ea2" (UID: "55775264-6e02-4184-ab8b-6cb8c29f0ea2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:27:13.432567 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.432397    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6" (OuterVolumeSpecName: "kube-api-access-w2qg6") pod "55775264-6e02-4184-ab8b-6cb8c29f0ea2" (UID: "55775264-6e02-4184-ab8b-6cb8c29f0ea2"). InnerVolumeSpecName "kube-api-access-w2qg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:27:13.530812 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.530782    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qg6\" (UniqueName: \"kubernetes.io/projected/55775264-6e02-4184-ab8b-6cb8c29f0ea2-kube-api-access-w2qg6\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:13.530812 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.530809    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55775264-6e02-4184-ab8b-6cb8c29f0ea2-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:13.530997 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.530823    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55775264-6e02-4184-ab8b-6cb8c29f0ea2-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:13.547014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.546986    2564 generic.go:356] "Generic (PLEG): container finished" podID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" containerID="2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374" exitCode=0
Jul 27 17:27:13.547155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.547043    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"
Jul 27 17:27:13.547155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.547064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" event={"ID":"55775264-6e02-4184-ab8b-6cb8c29f0ea2","Type":"ContainerDied","Data":"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"}
Jul 27 17:27:13.547155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.547119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm" event={"ID":"55775264-6e02-4184-ab8b-6cb8c29f0ea2","Type":"ContainerDied","Data":"c097c99be33a58777251b543207841d60f9747056c595419133171cb43348843"}
Jul 27 17:27:13.547155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.547143    2564 scope.go:117] "RemoveContainer" containerID="2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"
Jul 27 17:27:13.555340 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.555314    2564 scope.go:117] "RemoveContainer" containerID="2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"
Jul 27 17:27:13.555765 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:27:13.555744    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374\": container with ID starting with 2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374 not found: ID does not exist" containerID="2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"
Jul 27 17:27:13.555854 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.555772    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374"} err="failed to get container status \"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374\": rpc error: code = NotFound desc = could not find container \"2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374\": container with ID starting with 2b24b12c0062bd1a0ae8e73c1bf9e5e081c1ab437320b71a0306ec7145f69374 not found: ID does not exist"
Jul 27 17:27:13.576634 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.576223    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:27:13.581323 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:13.581295    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-41d9ef37-575bb7b8fd-b7tdm"]
Jul 27 17:27:14.658602 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:14.658568    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" path="/var/lib/kubelet/pods/55775264-6e02-4184-ab8b-6cb8c29f0ea2/volumes"
Jul 27 17:27:15.697817 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:15.697787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:27:21.548103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.548068    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:21.548598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.548578    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" containerName="maas-api"
Jul 27 17:27:21.548598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.548597    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" containerName="maas-api"
Jul 27 17:27:21.548715 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.548674    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="55775264-6e02-4184-ab8b-6cb8c29f0ea2" containerName="maas-api"
Jul 27 17:27:21.553852 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.553829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.557300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.557271    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:27:21.557435 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.557271    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-26960311-serving-cert\""
Jul 27 17:27:21.557435 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.557383    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:27:21.558651 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.558634    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:27:21.562758 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.562736    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:21.593139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.593118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.593245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.593144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxblb\" (UniqueName: \"kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.593245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.593166    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.694311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.694285    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.694311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.694312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxblb\" (UniqueName: \"kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.694461 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.694332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.694461 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:27:21.694417    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-serving-cert: secret "maas-api-e2e-migrate-26960311-serving-cert" not found
Jul 27 17:27:21.694537 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:27:21.694475    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls podName:ab2df269-b0b6-47eb-bbb7-5fe7aba79365 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:22.194456732 +0000 UTC m=+1862.026821687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls") pod "maas-api-e2e-migrate-26960311-f998cb458-29c9f" (UID: "ab2df269-b0b6-47eb-bbb7-5fe7aba79365") : secret "maas-api-e2e-migrate-26960311-serving-cert" not found
Jul 27 17:27:21.694838 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.694822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:21.716137 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:21.716118    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxblb\" (UniqueName: \"kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:22.198370 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.198336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:22.200726 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.200704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") pod \"maas-api-e2e-migrate-26960311-f998cb458-29c9f\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:22.266231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.266201    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:22.270223 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.270201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.274092 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.274072    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-26960311-dockercfg-lglrt\""
Jul 27 17:27:22.274348 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.274327    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-26960311\""
Jul 27 17:27:22.281220 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.281200    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:22.399706 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.399673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8m4\" (UniqueName: \"kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.399866 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.399738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.464766 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.464686    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:22.501063 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.501030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8m4\" (UniqueName: \"kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.501207 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.501097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.501753 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.501733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.512325 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.512304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8m4\" (UniqueName: \"kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4\") pod \"payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") " pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.581451 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.581426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:22.598995 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.598972    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:22.601103 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:27:22.601080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab2df269_b0b6_47eb_bbb7_5fe7aba79365.slice/crio-61a9e6fe0fac55053475525d0022d1bc784c3a65e071e831f38f185b9dedbded WatchSource:0}: Error finding container 61a9e6fe0fac55053475525d0022d1bc784c3a65e071e831f38f185b9dedbded: Status 404 returned error can't find the container with id 61a9e6fe0fac55053475525d0022d1bc784c3a65e071e831f38f185b9dedbded
Jul 27 17:27:22.715205 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:22.715130    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:22.718820 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:27:22.718790    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171aa9cb_3818_4812_9a41_457a62c3c03e.slice/crio-a6647a74831e2dcac16144e1b39238586d8b2c06d1ec7cad652fbce7c4239410 WatchSource:0}: Error finding container a6647a74831e2dcac16144e1b39238586d8b2c06d1ec7cad652fbce7c4239410: Status 404 returned error can't find the container with id a6647a74831e2dcac16144e1b39238586d8b2c06d1ec7cad652fbce7c4239410
Jul 27 17:27:23.594926 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:23.594818    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" event={"ID":"171aa9cb-3818-4812-9a41-457a62c3c03e","Type":"ContainerStarted","Data":"a6647a74831e2dcac16144e1b39238586d8b2c06d1ec7cad652fbce7c4239410"}
Jul 27 17:27:23.596640 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:23.596600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" event={"ID":"ab2df269-b0b6-47eb-bbb7-5fe7aba79365","Type":"ContainerStarted","Data":"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"}
Jul 27 17:27:23.596789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:23.596762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:23.596911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:23.596797    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" event={"ID":"ab2df269-b0b6-47eb-bbb7-5fe7aba79365","Type":"ContainerStarted","Data":"61a9e6fe0fac55053475525d0022d1bc784c3a65e071e831f38f185b9dedbded"}
Jul 27 17:27:23.618674 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:23.618378    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" podStartSLOduration=2.072666533 podStartE2EDuration="2.61835869s" totalImagesPullingTime="545.692157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:23.616583467 +0000 UTC m=+1863.448948467" watchObservedRunningTime="2026-07-27 17:27:23.61835869 +0000 UTC m=+1863.450723666"
Jul 27 17:27:24.602736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:24.602705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" event={"ID":"171aa9cb-3818-4812-9a41-457a62c3c03e","Type":"ContainerStarted","Data":"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"}
Jul 27 17:27:24.603089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:24.602899    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:24.623300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:24.623259    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" podStartSLOduration=0.810426599 podStartE2EDuration="2.623244213s" totalImagesPullingTime="1.812817614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:24.622298722 +0000 UTC m=+1864.454663711" watchObservedRunningTime="2026-07-27 17:27:24.623244213 +0000 UTC m=+1864.455609189"
Jul 27 17:27:28.991411 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:28.991373    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:27:29.607865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:29.607836    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:35.608170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:35.608139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:56.959734 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:56.959702    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:56.960253 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:56.959912    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" podUID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" containerName="maas-api" containerID="cri-o://d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3" gracePeriod=30
Jul 27 17:27:57.027720 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.027690    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:57.027942 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.027915    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" podUID="171aa9cb-3818-4812-9a41-457a62c3c03e" containerName="payload-processing" containerID="cri-o://8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723" gracePeriod=30
Jul 27 17:27:57.269236 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.269209    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:57.296249 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.296226    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:57.368806 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.368777    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") pod \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") "
Jul 27 17:27:57.368961 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.368809    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca\") pod \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") "
Jul 27 17:27:57.368961 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.368912    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxblb\" (UniqueName: \"kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb\") pod \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\" (UID: \"ab2df269-b0b6-47eb-bbb7-5fe7aba79365\") "
Jul 27 17:27:57.369239 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.369210    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab2df269-b0b6-47eb-bbb7-5fe7aba79365" (UID: "ab2df269-b0b6-47eb-bbb7-5fe7aba79365"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:27:57.370872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.370852    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab2df269-b0b6-47eb-bbb7-5fe7aba79365" (UID: "ab2df269-b0b6-47eb-bbb7-5fe7aba79365"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:27:57.370967 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.370918    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb" (OuterVolumeSpecName: "kube-api-access-nxblb") pod "ab2df269-b0b6-47eb-bbb7-5fe7aba79365" (UID: "ab2df269-b0b6-47eb-bbb7-5fe7aba79365"). InnerVolumeSpecName "kube-api-access-nxblb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:27:57.470408 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470370    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume\") pod \"171aa9cb-3818-4812-9a41-457a62c3c03e\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") "
Jul 27 17:27:57.470538 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470422    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z8m4\" (UniqueName: \"kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4\") pod \"171aa9cb-3818-4812-9a41-457a62c3c03e\" (UID: \"171aa9cb-3818-4812-9a41-457a62c3c03e\") "
Jul 27 17:27:57.470580 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470539    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:57.470580 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470549    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:57.470580 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470558    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxblb\" (UniqueName: \"kubernetes.io/projected/ab2df269-b0b6-47eb-bbb7-5fe7aba79365-kube-api-access-nxblb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:57.470699 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.470679    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "171aa9cb-3818-4812-9a41-457a62c3c03e" (UID: "171aa9cb-3818-4812-9a41-457a62c3c03e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:27:57.472550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.472532    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4" (OuterVolumeSpecName: "kube-api-access-4z8m4") pod "171aa9cb-3818-4812-9a41-457a62c3c03e" (UID: "171aa9cb-3818-4812-9a41-457a62c3c03e"). InnerVolumeSpecName "kube-api-access-4z8m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:27:57.571475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.571392    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171aa9cb-3818-4812-9a41-457a62c3c03e-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:57.571475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.571421    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8m4\" (UniqueName: \"kubernetes.io/projected/171aa9cb-3818-4812-9a41-457a62c3c03e-kube-api-access-4z8m4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:27:57.745786 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.745749    2564 generic.go:356] "Generic (PLEG): container finished" podID="171aa9cb-3818-4812-9a41-457a62c3c03e" containerID="8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723" exitCode=0
Jul 27 17:27:57.745992 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.745782    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" event={"ID":"171aa9cb-3818-4812-9a41-457a62c3c03e","Type":"ContainerDied","Data":"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"}
Jul 27 17:27:57.745992 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.745807    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"
Jul 27 17:27:57.745992 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.745839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc" event={"ID":"171aa9cb-3818-4812-9a41-457a62c3c03e","Type":"ContainerDied","Data":"a6647a74831e2dcac16144e1b39238586d8b2c06d1ec7cad652fbce7c4239410"}
Jul 27 17:27:57.745992 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.745866    2564 scope.go:117] "RemoveContainer" containerID="8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"
Jul 27 17:27:57.747260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.747239    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" containerID="d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3" exitCode=0
Jul 27 17:27:57.747363 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.747288    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"
Jul 27 17:27:57.747363 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.747316    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" event={"ID":"ab2df269-b0b6-47eb-bbb7-5fe7aba79365","Type":"ContainerDied","Data":"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"}
Jul 27 17:27:57.747363 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.747355    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f" event={"ID":"ab2df269-b0b6-47eb-bbb7-5fe7aba79365","Type":"ContainerDied","Data":"61a9e6fe0fac55053475525d0022d1bc784c3a65e071e831f38f185b9dedbded"}
Jul 27 17:27:57.755586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.755515    2564 scope.go:117] "RemoveContainer" containerID="8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"
Jul 27 17:27:57.755805 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:27:57.755762    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723\": container with ID starting with 8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723 not found: ID does not exist" containerID="8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"
Jul 27 17:27:57.755805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.755785    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723"} err="failed to get container status \"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723\": rpc error: code = NotFound desc = could not find container \"8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723\": container with ID starting with 8972e9b77ccc9eaa3e5d25536deff94508594f73c95fac2349d35d1555be9723 not found: ID does not exist"
Jul 27 17:27:57.755953 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.755807    2564 scope.go:117] "RemoveContainer" containerID="d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"
Jul 27 17:27:57.763235 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.763220    2564 scope.go:117] "RemoveContainer" containerID="d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"
Jul 27 17:27:57.763450 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:27:57.763434    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3\": container with ID starting with d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3 not found: ID does not exist" containerID="d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"
Jul 27 17:27:57.763495 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.763455    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3"} err="failed to get container status \"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3\": rpc error: code = NotFound desc = could not find container \"d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3\": container with ID starting with d60f9bd9007e903275b9bb67db7b73295dab5c5b66a3752f9b6bdd2a484799e3 not found: ID does not exist"
Jul 27 17:27:57.773442 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.773421    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:57.777700 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.777681    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-26960311-f998cb458-29c9f"]
Jul 27 17:27:57.790665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.790644    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:57.796383 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:57.796363    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-26960311-7cf58df6d5-jk8rc"]
Jul 27 17:27:58.660003 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:58.659973    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171aa9cb-3818-4812-9a41-457a62c3c03e" path="/var/lib/kubelet/pods/171aa9cb-3818-4812-9a41-457a62c3c03e/volumes"
Jul 27 17:27:58.660349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:58.660325    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" path="/var/lib/kubelet/pods/ab2df269-b0b6-47eb-bbb7-5fe7aba79365/volumes"
Jul 27 17:27:59.690828 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:27:59.690798    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:28:05.138310 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138277    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138583    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" containerName="maas-api"
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" containerName="maas-api"
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138611    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171aa9cb-3818-4812-9a41-457a62c3c03e" containerName="payload-processing"
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138617    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="171aa9cb-3818-4812-9a41-457a62c3c03e" containerName="payload-processing"
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138671    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="171aa9cb-3818-4812-9a41-457a62c3c03e" containerName="payload-processing"
Jul 27 17:28:05.138680 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.138681    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab2df269-b0b6-47eb-bbb7-5fe7aba79365" containerName="maas-api"
Jul 27 17:28:05.143408 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.143390    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.148028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.148005    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:28:05.148169 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.148141    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:28:05.149347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.149326    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:28:05.149490 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.149352    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eda34c7f-serving-cert\""
Jul 27 17:28:05.160712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.160690    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:05.228710 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.228683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.228810 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.228713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxgf\" (UniqueName: \"kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.228810 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.228788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.329421 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.329401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.329521 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.329431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxgf\" (UniqueName: \"kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.329521 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.329456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.329620 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:05.329567    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-serving-cert: secret "maas-api-e2e-ait-eda34c7f-serving-cert" not found
Jul 27 17:28:05.329620 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:05.329619    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls podName:254bfba4-321d-42a2-91d4-28387ec7dd11 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:05.829604894 +0000 UTC m=+1905.661969848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls") pod "maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" (UID: "254bfba4-321d-42a2-91d4-28387ec7dd11") : secret "maas-api-e2e-ait-eda34c7f-serving-cert" not found
Jul 27 17:28:05.330095 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.330079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.344041 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.344023    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxgf\" (UniqueName: \"kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.832350 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.832321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:05.834494 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:05.834470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") pod \"maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:06.001006 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.000977    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:06.004265 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.004249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.007902 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.007860    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eda34c7f\""
Jul 27 17:28:06.019215 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.019197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:06.034285 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.034258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79n8m\" (UniqueName: \"kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.034385 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.034289    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.053931 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.053907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:06.077667 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.076651    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:06.085211 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.085157    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.089119 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.089098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eda34c7f-dockercfg-mcksx\""
Jul 27 17:28:06.100479 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.100458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:06.135069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.134818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwqb9\" (UniqueName: \"kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.135069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.134973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79n8m\" (UniqueName: \"kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.135069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.135003    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.135069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.135033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.135954 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.135931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.153759 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.153727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79n8m\" (UniqueName: \"kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m\") pod \"payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") " pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.214922 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.214899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:06.216972 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:06.216941    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254bfba4_321d_42a2_91d4_28387ec7dd11.slice/crio-6f99250fad552d8d6cae335a475e6e5bfeaf1b868e297c885cc1e79614ab7899 WatchSource:0}: Error finding container 6f99250fad552d8d6cae335a475e6e5bfeaf1b868e297c885cc1e79614ab7899: Status 404 returned error can't find the container with id 6f99250fad552d8d6cae335a475e6e5bfeaf1b868e297c885cc1e79614ab7899
Jul 27 17:28:06.236281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.236259    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.236385 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.236296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwqb9\" (UniqueName: \"kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.236915 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.236898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.246133 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.246115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwqb9\" (UniqueName: \"kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9\") pod \"payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.314633 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.314610    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.397588 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.397559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.469174 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.469151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:06.470796 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:06.470768    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73406b4_57ff_46f6_ab70_176b8bdfeb88.slice/crio-d51a92364b2d6242fd0df8a0e33866cb0392a03c173a912402a00ed2d5341a40 WatchSource:0}: Error finding container d51a92364b2d6242fd0df8a0e33866cb0392a03c173a912402a00ed2d5341a40: Status 404 returned error can't find the container with id d51a92364b2d6242fd0df8a0e33866cb0392a03c173a912402a00ed2d5341a40
Jul 27 17:28:06.541241 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.541215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:06.545283 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:06.545261    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656d3819_1df2_4195_b706_78e2b2b99bcc.slice/crio-a01bb2d529e320c11f4e311d7f94014116f057242c874173d4e4e2751fb57880 WatchSource:0}: Error finding container a01bb2d529e320c11f4e311d7f94014116f057242c874173d4e4e2751fb57880: Status 404 returned error can't find the container with id a01bb2d529e320c11f4e311d7f94014116f057242c874173d4e4e2751fb57880
Jul 27 17:28:06.790107 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.790073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" event={"ID":"254bfba4-321d-42a2-91d4-28387ec7dd11","Type":"ContainerStarted","Data":"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"}
Jul 27 17:28:06.790285 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.790122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" event={"ID":"254bfba4-321d-42a2-91d4-28387ec7dd11","Type":"ContainerStarted","Data":"6f99250fad552d8d6cae335a475e6e5bfeaf1b868e297c885cc1e79614ab7899"}
Jul 27 17:28:06.790285 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.790160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:06.791677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.791651    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" event={"ID":"656d3819-1df2-4195-b706-78e2b2b99bcc","Type":"ContainerStarted","Data":"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"}
Jul 27 17:28:06.791798 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.791697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" event={"ID":"656d3819-1df2-4195-b706-78e2b2b99bcc","Type":"ContainerStarted","Data":"a01bb2d529e320c11f4e311d7f94014116f057242c874173d4e4e2751fb57880"}
Jul 27 17:28:06.791798 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.791727    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:06.793065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.793044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" event={"ID":"e73406b4-57ff-46f6-ab70-176b8bdfeb88","Type":"ContainerStarted","Data":"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"}
Jul 27 17:28:06.793147 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.793084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" event={"ID":"e73406b4-57ff-46f6-ab70-176b8bdfeb88","Type":"ContainerStarted","Data":"d51a92364b2d6242fd0df8a0e33866cb0392a03c173a912402a00ed2d5341a40"}
Jul 27 17:28:06.793195 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.793162    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:06.834032 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.833986    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" podStartSLOduration=1.504786551 podStartE2EDuration="1.833969362s" totalImagesPullingTime="329.182811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:06.832017592 +0000 UTC m=+1906.664382596" watchObservedRunningTime="2026-07-27 17:28:06.833969362 +0000 UTC m=+1906.666334340"
Jul 27 17:28:06.860052 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:06.860005    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" podStartSLOduration=0.859993128 podStartE2EDuration="859.993128ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:06.858985413 +0000 UTC m=+1906.691350389" watchObservedRunningTime="2026-07-27 17:28:06.859993128 +0000 UTC m=+1906.692358104"
Jul 27 17:28:12.804034 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:12.804004    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:13.988868 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:13.988812    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" podStartSLOduration=8.988796962 podStartE2EDuration="8.988796962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:06.886817462 +0000 UTC m=+1906.719182439" watchObservedRunningTime="2026-07-27 17:28:13.988796962 +0000 UTC m=+1913.821161938"
Jul 27 17:28:14.920005 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:14.919976    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:28:17.805704 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:17.801096    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:17.806187 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:17.805859    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:42.096887 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.096844    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:42.097375 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.097046    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" podUID="254bfba4-321d-42a2-91d4-28387ec7dd11" containerName="maas-api" containerID="cri-o://1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9" gracePeriod=30
Jul 27 17:28:42.164819 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.164785    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:42.165067 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.165036    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" podUID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" containerName="payload-processing" containerID="cri-o://5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262" gracePeriod=30
Jul 27 17:28:42.179660 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.179621    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:42.179913 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.179865    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" podUID="656d3819-1df2-4195-b706-78e2b2b99bcc" containerName="payload-pre-processing" containerID="cri-o://152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c" gracePeriod=30
Jul 27 17:28:42.435077 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.435047    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:42.462498 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.462474    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:42.471284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.471262    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:42.521428 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.521396    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume\") pod \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") "
Jul 27 17:28:42.521598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.521468    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwqb9\" (UniqueName: \"kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9\") pod \"656d3819-1df2-4195-b706-78e2b2b99bcc\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") "
Jul 27 17:28:42.521598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.521521    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79n8m\" (UniqueName: \"kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m\") pod \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\" (UID: \"e73406b4-57ff-46f6-ab70-176b8bdfeb88\") "
Jul 27 17:28:42.521598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.521561    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume\") pod \"656d3819-1df2-4195-b706-78e2b2b99bcc\" (UID: \"656d3819-1df2-4195-b706-78e2b2b99bcc\") "
Jul 27 17:28:42.521787 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.521731    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e73406b4-57ff-46f6-ab70-176b8bdfeb88" (UID: "e73406b4-57ff-46f6-ab70-176b8bdfeb88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:28:42.522068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.522042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "656d3819-1df2-4195-b706-78e2b2b99bcc" (UID: "656d3819-1df2-4195-b706-78e2b2b99bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:28:42.523505 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.523482    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m" (OuterVolumeSpecName: "kube-api-access-79n8m") pod "e73406b4-57ff-46f6-ab70-176b8bdfeb88" (UID: "e73406b4-57ff-46f6-ab70-176b8bdfeb88"). InnerVolumeSpecName "kube-api-access-79n8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:28:42.523627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.523509    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9" (OuterVolumeSpecName: "kube-api-access-pwqb9") pod "656d3819-1df2-4195-b706-78e2b2b99bcc" (UID: "656d3819-1df2-4195-b706-78e2b2b99bcc"). InnerVolumeSpecName "kube-api-access-pwqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:28:42.622120 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622092    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca\") pod \"254bfba4-321d-42a2-91d4-28387ec7dd11\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") "
Jul 27 17:28:42.622260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622132    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hxgf\" (UniqueName: \"kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf\") pod \"254bfba4-321d-42a2-91d4-28387ec7dd11\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") "
Jul 27 17:28:42.622260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622169    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") pod \"254bfba4-321d-42a2-91d4-28387ec7dd11\" (UID: \"254bfba4-321d-42a2-91d4-28387ec7dd11\") "
Jul 27 17:28:42.622362 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622301    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/656d3819-1df2-4195-b706-78e2b2b99bcc-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.622362 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622313    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e73406b4-57ff-46f6-ab70-176b8bdfeb88-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.622362 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622321    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwqb9\" (UniqueName: \"kubernetes.io/projected/656d3819-1df2-4195-b706-78e2b2b99bcc-kube-api-access-pwqb9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.622362 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622331    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79n8m\" (UniqueName: \"kubernetes.io/projected/e73406b4-57ff-46f6-ab70-176b8bdfeb88-kube-api-access-79n8m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.622611 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.622388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "254bfba4-321d-42a2-91d4-28387ec7dd11" (UID: "254bfba4-321d-42a2-91d4-28387ec7dd11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:28:42.624089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.624068    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "254bfba4-321d-42a2-91d4-28387ec7dd11" (UID: "254bfba4-321d-42a2-91d4-28387ec7dd11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:28:42.624167 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.624131    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf" (OuterVolumeSpecName: "kube-api-access-4hxgf") pod "254bfba4-321d-42a2-91d4-28387ec7dd11" (UID: "254bfba4-321d-42a2-91d4-28387ec7dd11"). InnerVolumeSpecName "kube-api-access-4hxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:28:42.722740 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.722713    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254bfba4-321d-42a2-91d4-28387ec7dd11-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.722740 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.722736    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hxgf\" (UniqueName: \"kubernetes.io/projected/254bfba4-321d-42a2-91d4-28387ec7dd11-kube-api-access-4hxgf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.722740 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.722746    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254bfba4-321d-42a2-91d4-28387ec7dd11-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:28:42.950640 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.950548    2564 generic.go:356] "Generic (PLEG): container finished" podID="254bfba4-321d-42a2-91d4-28387ec7dd11" containerID="1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9" exitCode=0
Jul 27 17:28:42.950640 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.950598    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"
Jul 27 17:28:42.950906 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.950632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" event={"ID":"254bfba4-321d-42a2-91d4-28387ec7dd11","Type":"ContainerDied","Data":"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"}
Jul 27 17:28:42.950906 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.950690    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2" event={"ID":"254bfba4-321d-42a2-91d4-28387ec7dd11","Type":"ContainerDied","Data":"6f99250fad552d8d6cae335a475e6e5bfeaf1b868e297c885cc1e79614ab7899"}
Jul 27 17:28:42.950906 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.950713    2564 scope.go:117] "RemoveContainer" containerID="1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"
Jul 27 17:28:42.951972 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.951951    2564 generic.go:356] "Generic (PLEG): container finished" podID="656d3819-1df2-4195-b706-78e2b2b99bcc" containerID="152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c" exitCode=0
Jul 27 17:28:42.952086 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.952008    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"
Jul 27 17:28:42.952086 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.952043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" event={"ID":"656d3819-1df2-4195-b706-78e2b2b99bcc","Type":"ContainerDied","Data":"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"}
Jul 27 17:28:42.952201 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.952093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n" event={"ID":"656d3819-1df2-4195-b706-78e2b2b99bcc","Type":"ContainerDied","Data":"a01bb2d529e320c11f4e311d7f94014116f057242c874173d4e4e2751fb57880"}
Jul 27 17:28:42.953524 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.953502    2564 generic.go:356] "Generic (PLEG): container finished" podID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" containerID="5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262" exitCode=0
Jul 27 17:28:42.953603 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.953535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" event={"ID":"e73406b4-57ff-46f6-ab70-176b8bdfeb88","Type":"ContainerDied","Data":"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"}
Jul 27 17:28:42.953603 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.953571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck" event={"ID":"e73406b4-57ff-46f6-ab70-176b8bdfeb88","Type":"ContainerDied","Data":"d51a92364b2d6242fd0df8a0e33866cb0392a03c173a912402a00ed2d5341a40"}
Jul 27 17:28:42.953603 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.953539    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"
Jul 27 17:28:42.960268 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.960251    2564 scope.go:117] "RemoveContainer" containerID="1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"
Jul 27 17:28:42.960512 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:42.960495    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9\": container with ID starting with 1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9 not found: ID does not exist" containerID="1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"
Jul 27 17:28:42.960558 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.960522    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9"} err="failed to get container status \"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9\": rpc error: code = NotFound desc = could not find container \"1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9\": container with ID starting with 1438c77e53b8ef0be5cfe2b2e2550c37c30e079b6fca726938d180ff263648e9 not found: ID does not exist"
Jul 27 17:28:42.960558 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.960539    2564 scope.go:117] "RemoveContainer" containerID="152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"
Jul 27 17:28:42.967928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.967910    2564 scope.go:117] "RemoveContainer" containerID="152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"
Jul 27 17:28:42.968157 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:42.968140    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c\": container with ID starting with 152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c not found: ID does not exist" containerID="152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"
Jul 27 17:28:42.968206 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.968165    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c"} err="failed to get container status \"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c\": rpc error: code = NotFound desc = could not find container \"152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c\": container with ID starting with 152cb9af8107bc3ad4ca569f5ba6c0f22b30a27957ef95deb5a7299d25fa797c not found: ID does not exist"
Jul 27 17:28:42.968206 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.968181    2564 scope.go:117] "RemoveContainer" containerID="5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"
Jul 27 17:28:42.974909 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.974862    2564 scope.go:117] "RemoveContainer" containerID="5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"
Jul 27 17:28:42.975136 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:42.975117    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262\": container with ID starting with 5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262 not found: ID does not exist" containerID="5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"
Jul 27 17:28:42.975188 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.975144    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262"} err="failed to get container status \"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262\": rpc error: code = NotFound desc = could not find container \"5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262\": container with ID starting with 5978dc0de740c15e16b7cfd0d85e5ae409d9bc156d579e59b078fcdadbd7e262 not found: ID does not exist"
Jul 27 17:28:42.989088 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.989069    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:42.996467 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:42.996448    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eda34c7f-86c4658dbb-wrmz2"]
Jul 27 17:28:43.008665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:43.008647    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:43.014974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:43.014956    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eda34c7f-8bbb5c9fb-qwt9n"]
Jul 27 17:28:43.027092 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:43.027071    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:43.036354 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:43.036336    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eda34c7f-68bd577798-mzwck"]
Jul 27 17:28:44.657644 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:44.657611    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254bfba4-321d-42a2-91d4-28387ec7dd11" path="/var/lib/kubelet/pods/254bfba4-321d-42a2-91d4-28387ec7dd11/volumes"
Jul 27 17:28:44.658005 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:44.657981    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656d3819-1df2-4195-b706-78e2b2b99bcc" path="/var/lib/kubelet/pods/656d3819-1df2-4195-b706-78e2b2b99bcc/volumes"
Jul 27 17:28:44.658295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:44.658281    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" path="/var/lib/kubelet/pods/e73406b4-57ff-46f6-ab70-176b8bdfeb88/volumes"
Jul 27 17:28:44.993989 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:44.993959    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:28:50.872576 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872541    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872869    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" containerName="payload-processing"
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872894    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" containerName="payload-processing"
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872919    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656d3819-1df2-4195-b706-78e2b2b99bcc" containerName="payload-pre-processing"
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872930    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="656d3819-1df2-4195-b706-78e2b2b99bcc" containerName="payload-pre-processing"
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872944    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254bfba4-321d-42a2-91d4-28387ec7dd11" containerName="maas-api"
Jul 27 17:28:50.872952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.872950    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="254bfba4-321d-42a2-91d4-28387ec7dd11" containerName="maas-api"
Jul 27 17:28:50.873152 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.873013    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="656d3819-1df2-4195-b706-78e2b2b99bcc" containerName="payload-pre-processing"
Jul 27 17:28:50.873152 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.873022    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="254bfba4-321d-42a2-91d4-28387ec7dd11" containerName="maas-api"
Jul 27 17:28:50.873152 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.873031    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73406b4-57ff-46f6-ab70-176b8bdfeb88" containerName="payload-processing"
Jul 27 17:28:50.877615 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.877599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:50.882276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.882251    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:28:50.882471 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.882453    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-834e32af-serving-cert\""
Jul 27 17:28:50.882678 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.882663    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:28:50.883701 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.883677    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:28:50.886813 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.886787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:28:50.990868 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.990842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:50.991030 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.990874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:50.991030 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:50.990924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6459x\" (UniqueName: \"kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.092218 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.092197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.092338 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.092227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.092338 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.092257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6459x\" (UniqueName: \"kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.092338 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:51.092329    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-serving-cert: secret "maas-api-e2e-derive-834e32af-serving-cert" not found
Jul 27 17:28:51.092510 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:28:51.092388    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls podName:086d7eca-d1fc-4bf7-98d6-b190a1a43a5f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:51.592373363 +0000 UTC m=+1951.424738317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls") pod "maas-api-e2e-derive-834e32af-54f8676688-fml29" (UID: "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f") : secret "maas-api-e2e-derive-834e32af-serving-cert" not found
Jul 27 17:28:51.092932 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.092911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.104563 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.104543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6459x\" (UniqueName: \"kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.595519 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.595481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.597985 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.597962    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") pod \"maas-api-e2e-derive-834e32af-54f8676688-fml29\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.606233 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.606211    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:28:51.609441 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.609426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.612600 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.612577    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-834e32af\""
Jul 27 17:28:51.619805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.619784    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:28:51.621852 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.621834    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:28:51.625003 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.624986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.628073 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.628046    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-834e32af-dockercfg-g454b\""
Jul 27 17:28:51.635619 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.635597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:28:51.696651 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.696626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p99s\" (UniqueName: \"kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.696795 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.696663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.788593 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.788564    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:51.797524 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.797503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.797619 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.797582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p99s\" (UniqueName: \"kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.797619 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.797602    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.797723 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.797620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pdl\" (UniqueName: \"kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.798269 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.798247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.808388 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.808367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p99s\" (UniqueName: \"kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s\") pod \"payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.898454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.898422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.898938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.898503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pdl\" (UniqueName: \"kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.899125 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.899104    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.914218 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.914195    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pdl\" (UniqueName: \"kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl\") pod \"payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") " pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.920452 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.920405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:51.921521 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.921494    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:28:51.923346 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:51.923324    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086d7eca_d1fc_4bf7_98d6_b190a1a43a5f.slice/crio-31534b3bede31983e8056cf4d93afc3e8635605f04ed658e66e7345db799f1b2 WatchSource:0}: Error finding container 31534b3bede31983e8056cf4d93afc3e8635605f04ed658e66e7345db799f1b2: Status 404 returned error can't find the container with id 31534b3bede31983e8056cf4d93afc3e8635605f04ed658e66e7345db799f1b2
Jul 27 17:28:51.936000 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.935980    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:51.996696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:51.996642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" event={"ID":"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f","Type":"ContainerStarted","Data":"31534b3bede31983e8056cf4d93afc3e8635605f04ed658e66e7345db799f1b2"}
Jul 27 17:28:52.058436 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:52.058408    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:28:52.062025 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:52.061999    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode547847d_e7c9_4591_883c_e91407202944.slice/crio-6b1640cc342de591d424cd44876f53d33a595f332c47be998fc76b95d720a6d8 WatchSource:0}: Error finding container 6b1640cc342de591d424cd44876f53d33a595f332c47be998fc76b95d720a6d8: Status 404 returned error can't find the container with id 6b1640cc342de591d424cd44876f53d33a595f332c47be998fc76b95d720a6d8
Jul 27 17:28:52.087655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:52.087626    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:28:52.091845 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:28:52.091819    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da4509_7260_4d22_916f_8d0d97b87eec.slice/crio-405b76304933c4c81a5e9c0512119503e05ba8f29e9300733eb2a77cab6718dd WatchSource:0}: Error finding container 405b76304933c4c81a5e9c0512119503e05ba8f29e9300733eb2a77cab6718dd: Status 404 returned error can't find the container with id 405b76304933c4c81a5e9c0512119503e05ba8f29e9300733eb2a77cab6718dd
Jul 27 17:28:53.003234 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.003194    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" event={"ID":"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f","Type":"ContainerStarted","Data":"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"}
Jul 27 17:28:53.003712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.003271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:28:53.004670 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.004649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" event={"ID":"d8da4509-7260-4d22-916f-8d0d97b87eec","Type":"ContainerStarted","Data":"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"}
Jul 27 17:28:53.004796 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.004693    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" event={"ID":"d8da4509-7260-4d22-916f-8d0d97b87eec","Type":"ContainerStarted","Data":"405b76304933c4c81a5e9c0512119503e05ba8f29e9300733eb2a77cab6718dd"}
Jul 27 17:28:53.004796 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.004721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:28:53.006063 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.006043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" event={"ID":"e547847d-e7c9-4591-883c-e91407202944","Type":"ContainerStarted","Data":"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"}
Jul 27 17:28:53.006160 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.006077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" event={"ID":"e547847d-e7c9-4591-883c-e91407202944","Type":"ContainerStarted","Data":"6b1640cc342de591d424cd44876f53d33a595f332c47be998fc76b95d720a6d8"}
Jul 27 17:28:53.006226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.006158    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:28:53.031193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.031144    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" podStartSLOduration=2.477557214 podStartE2EDuration="3.03112546s" totalImagesPullingTime="553.568246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:53.029539829 +0000 UTC m=+1952.861904805" watchObservedRunningTime="2026-07-27 17:28:53.03112546 +0000 UTC m=+1952.863490440"
Jul 27 17:28:53.094696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:53.094661    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" podStartSLOduration=2.094648366 podStartE2EDuration="2.094648366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:53.054338883 +0000 UTC m=+1952.886703855" watchObservedRunningTime="2026-07-27 17:28:53.094648366 +0000 UTC m=+1952.927013341"
Jul 27 17:28:57.040512 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:57.040458    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" podStartSLOduration=6.040441416 podStartE2EDuration="6.040441416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:53.096041017 +0000 UTC m=+1952.928405993" watchObservedRunningTime="2026-07-27 17:28:57.040441416 +0000 UTC m=+1956.872806399"
Jul 27 17:28:57.040965 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:57.040736    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:28:59.016374 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:28:59.016345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:29:04.013393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:04.013351    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:29:04.013754 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:04.013432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:29:26.083949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.083919    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:29:26.084432 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.084127    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" podUID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" containerName="maas-api" containerID="cri-o://e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37" gracePeriod=30
Jul 27 17:29:26.145737 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.145705    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:29:26.146432 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.145966    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" podUID="d8da4509-7260-4d22-916f-8d0d97b87eec" containerName="payload-processing" containerID="cri-o://c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2" gracePeriod=30
Jul 27 17:29:26.154663 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.154626    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:29:26.154872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.154843    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" podUID="e547847d-e7c9-4591-883c-e91407202944" containerName="payload-pre-processing" containerID="cri-o://ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53" gracePeriod=30
Jul 27 17:29:26.442423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.442032    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:29:26.452625 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.452601    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:29:26.465466 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.465444    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:29:26.466955 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.466933    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") pod \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") "
Jul 27 17:29:26.467056 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.466961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca\") pod \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") "
Jul 27 17:29:26.467056 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.467011    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6459x\" (UniqueName: \"kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x\") pod \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\" (UID: \"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f\") "
Jul 27 17:29:26.467269 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.467248    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" (UID: "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:29:26.469326 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.469260    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x" (OuterVolumeSpecName: "kube-api-access-6459x") pod "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" (UID: "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f"). InnerVolumeSpecName "kube-api-access-6459x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:29:26.469452 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.469258    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" (UID: "086d7eca-d1fc-4bf7-98d6-b190a1a43a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:29:26.568232 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568204    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4pdl\" (UniqueName: \"kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl\") pod \"d8da4509-7260-4d22-916f-8d0d97b87eec\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") "
Jul 27 17:29:26.568377 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568238    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume\") pod \"d8da4509-7260-4d22-916f-8d0d97b87eec\" (UID: \"d8da4509-7260-4d22-916f-8d0d97b87eec\") "
Jul 27 17:29:26.568377 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p99s\" (UniqueName: \"kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s\") pod \"e547847d-e7c9-4591-883c-e91407202944\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") "
Jul 27 17:29:26.568377 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568318    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume\") pod \"e547847d-e7c9-4591-883c-e91407202944\" (UID: \"e547847d-e7c9-4591-883c-e91407202944\") "
Jul 27 17:29:26.568595 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568580    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.568660 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568602    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.568660 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568616    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6459x\" (UniqueName: \"kubernetes.io/projected/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f-kube-api-access-6459x\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.568755 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568662    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8da4509-7260-4d22-916f-8d0d97b87eec" (UID: "d8da4509-7260-4d22-916f-8d0d97b87eec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:29:26.568755 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.568707    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e547847d-e7c9-4591-883c-e91407202944" (UID: "e547847d-e7c9-4591-883c-e91407202944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:29:26.572248 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.570527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s" (OuterVolumeSpecName: "kube-api-access-6p99s") pod "e547847d-e7c9-4591-883c-e91407202944" (UID: "e547847d-e7c9-4591-883c-e91407202944"). InnerVolumeSpecName "kube-api-access-6p99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:29:26.572248 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.570652    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl" (OuterVolumeSpecName: "kube-api-access-m4pdl") pod "d8da4509-7260-4d22-916f-8d0d97b87eec" (UID: "d8da4509-7260-4d22-916f-8d0d97b87eec"). InnerVolumeSpecName "kube-api-access-m4pdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:29:26.669077 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.669059    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p99s\" (UniqueName: \"kubernetes.io/projected/e547847d-e7c9-4591-883c-e91407202944-kube-api-access-6p99s\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.669177 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.669078    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e547847d-e7c9-4591-883c-e91407202944-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.669177 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.669088    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pdl\" (UniqueName: \"kubernetes.io/projected/d8da4509-7260-4d22-916f-8d0d97b87eec-kube-api-access-m4pdl\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:26.669177 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:26.669096    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8da4509-7260-4d22-916f-8d0d97b87eec-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:29:27.172734 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.172683    2564 generic.go:356] "Generic (PLEG): container finished" podID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" containerID="e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37" exitCode=0
Jul 27 17:29:27.173200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.172747    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"
Jul 27 17:29:27.173200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.172776    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" event={"ID":"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f","Type":"ContainerDied","Data":"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"}
Jul 27 17:29:27.173200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.172837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29" event={"ID":"086d7eca-d1fc-4bf7-98d6-b190a1a43a5f","Type":"ContainerDied","Data":"31534b3bede31983e8056cf4d93afc3e8635605f04ed658e66e7345db799f1b2"}
Jul 27 17:29:27.173200 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.172874    2564 scope.go:117] "RemoveContainer" containerID="e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"
Jul 27 17:29:27.174295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.174277    2564 generic.go:356] "Generic (PLEG): container finished" podID="d8da4509-7260-4d22-916f-8d0d97b87eec" containerID="c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2" exitCode=0
Jul 27 17:29:27.174384 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.174355    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"
Jul 27 17:29:27.174384 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.174365    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" event={"ID":"d8da4509-7260-4d22-916f-8d0d97b87eec","Type":"ContainerDied","Data":"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"}
Jul 27 17:29:27.174484 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.174417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46" event={"ID":"d8da4509-7260-4d22-916f-8d0d97b87eec","Type":"ContainerDied","Data":"405b76304933c4c81a5e9c0512119503e05ba8f29e9300733eb2a77cab6718dd"}
Jul 27 17:29:27.175866 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.175843    2564 generic.go:356] "Generic (PLEG): container finished" podID="e547847d-e7c9-4591-883c-e91407202944" containerID="ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53" exitCode=0
Jul 27 17:29:27.175990 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.175910    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"
Jul 27 17:29:27.175990 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.175926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" event={"ID":"e547847d-e7c9-4591-883c-e91407202944","Type":"ContainerDied","Data":"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"}
Jul 27 17:29:27.175990 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.175977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b" event={"ID":"e547847d-e7c9-4591-883c-e91407202944","Type":"ContainerDied","Data":"6b1640cc342de591d424cd44876f53d33a595f332c47be998fc76b95d720a6d8"}
Jul 27 17:29:27.181934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.181915    2564 scope.go:117] "RemoveContainer" containerID="e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"
Jul 27 17:29:27.182200 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:29:27.182178    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37\": container with ID starting with e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37 not found: ID does not exist" containerID="e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"
Jul 27 17:29:27.182289 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.182206    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37"} err="failed to get container status \"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37\": rpc error: code = NotFound desc = could not find container \"e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37\": container with ID starting with e2dcfbfd711e4dad513b1dc3a4aeb3c5980b7a5ba8c2d24ed89cc8afe1740c37 not found: ID does not exist"
Jul 27 17:29:27.182289 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.182221    2564 scope.go:117] "RemoveContainer" containerID="c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"
Jul 27 17:29:27.189389 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.189374    2564 scope.go:117] "RemoveContainer" containerID="c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"
Jul 27 17:29:27.189630 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:29:27.189613    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2\": container with ID starting with c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2 not found: ID does not exist" containerID="c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"
Jul 27 17:29:27.189686 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.189635    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2"} err="failed to get container status \"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2\": rpc error: code = NotFound desc = could not find container \"c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2\": container with ID starting with c9e39c526a33d2bad9c5879e04cbd3855114747842090f2588b1a4e5f92972d2 not found: ID does not exist"
Jul 27 17:29:27.189686 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.189651    2564 scope.go:117] "RemoveContainer" containerID="ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"
Jul 27 17:29:27.197479 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.197461    2564 scope.go:117] "RemoveContainer" containerID="ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"
Jul 27 17:29:27.197730 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:29:27.197709    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53\": container with ID starting with ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53 not found: ID does not exist" containerID="ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"
Jul 27 17:29:27.197791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.197735    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53"} err="failed to get container status \"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53\": rpc error: code = NotFound desc = could not find container \"ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53\": container with ID starting with ba3de8a650e158054e224c902f6d34f31d75211e1229f59aeb7f69709424da53 not found: ID does not exist"
Jul 27 17:29:27.198775 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.198756    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:29:27.203967 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.203947    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-834e32af-54f8676688-fml29"]
Jul 27 17:29:27.216637 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.216616    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:29:27.218083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.218066    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-834e32af-cfcdf88fb-ssc9b"]
Jul 27 17:29:27.230551 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.230531    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:29:27.235103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:27.235084    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-834e32af-75cd7d5f58-bjz46"]
Jul 27 17:29:28.658280 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:28.658243    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" path="/var/lib/kubelet/pods/086d7eca-d1fc-4bf7-98d6-b190a1a43a5f/volumes"
Jul 27 17:29:28.658763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:28.658741    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8da4509-7260-4d22-916f-8d0d97b87eec" path="/var/lib/kubelet/pods/d8da4509-7260-4d22-916f-8d0d97b87eec/volumes"
Jul 27 17:29:28.659139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:28.659119    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e547847d-e7c9-4591-883c-e91407202944" path="/var/lib/kubelet/pods/e547847d-e7c9-4591-883c-e91407202944/volumes"
Jul 27 17:29:28.893757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:29:28.893730    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:30:59.192223 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:30:59.192185    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:31:07.388610 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:07.388578    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:31:14.383156 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383118    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383436    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8da4509-7260-4d22-916f-8d0d97b87eec" containerName="payload-processing"
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383447    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8da4509-7260-4d22-916f-8d0d97b87eec" containerName="payload-processing"
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383456    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" containerName="maas-api"
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383462    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" containerName="maas-api"
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383470    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e547847d-e7c9-4591-883c-e91407202944" containerName="payload-pre-processing"
Jul 27 17:31:14.383514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383477    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e547847d-e7c9-4591-883c-e91407202944" containerName="payload-pre-processing"
Jul 27 17:31:14.383716 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383536    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e547847d-e7c9-4591-883c-e91407202944" containerName="payload-pre-processing"
Jul 27 17:31:14.383716 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383546    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8da4509-7260-4d22-916f-8d0d97b87eec" containerName="payload-processing"
Jul 27 17:31:14.383716 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.383555    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="086d7eca-d1fc-4bf7-98d6-b190a1a43a5f" containerName="maas-api"
Jul 27 17:31:14.386512 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.386493    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.389318 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.389295    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:31:14.389429 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.389357    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:31:14.390768 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.390748    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-18117344-serving-cert\""
Jul 27 17:31:14.390874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.390775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:31:14.404341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.404315    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:14.424245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.424206    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587cj\" (UniqueName: \"kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.424365 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.424348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.424416 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.424382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.525170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.525138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.525170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.525170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.525391 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.525210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587cj\" (UniqueName: \"kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.525391 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:31:14.525313    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-18117344-serving-cert: secret "maas-api-e2e-mt-18117344-serving-cert" not found
Jul 27 17:31:14.525391 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:31:14.525373    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls podName:424cb238-c952-43a9-bc11-61549235c627 nodeName:}" failed. No retries permitted until 2026-07-27 17:31:15.025356379 +0000 UTC m=+2094.857721334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls") pod "maas-api-e2e-mt-18117344-59b585cd59-kjmjx" (UID: "424cb238-c952-43a9-bc11-61549235c627") : secret "maas-api-e2e-mt-18117344-serving-cert" not found
Jul 27 17:31:14.525935 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.525868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:14.538262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:14.538238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-587cj\" (UniqueName: \"kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:15.029682 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.029650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:15.032149 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.032116    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") pod \"maas-api-e2e-mt-18117344-59b585cd59-kjmjx\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:15.297136 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.297053    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:15.335571 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.335540    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:15.339298 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.339278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.342737 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.342717    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18117344-dockercfg-66wrl\""
Jul 27 17:31:15.345091 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.345075    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18117344\""
Jul 27 17:31:15.355902 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.355862    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:15.432999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.432970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b46vm\" (UniqueName: \"kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.433325 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.433008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.454782 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.454330    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:15.457692 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:31:15.457436    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424cb238_c952_43a9_bc11_61549235c627.slice/crio-0c3b5c155da97f6ffc83447984ec9a9e539050b86bcd7d0b75682e4cbe7563ee WatchSource:0}: Error finding container 0c3b5c155da97f6ffc83447984ec9a9e539050b86bcd7d0b75682e4cbe7563ee: Status 404 returned error can't find the container with id 0c3b5c155da97f6ffc83447984ec9a9e539050b86bcd7d0b75682e4cbe7563ee
Jul 27 17:31:15.534072 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.534047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b46vm\" (UniqueName: \"kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.534186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.534079    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.534710 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.534692    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.548587 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.548535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b46vm\" (UniqueName: \"kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm\") pod \"payload-processing-e2e-mt-18117344-5bc7979646-cvtxn\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") " pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.653258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.653236    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:15.660299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.660269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" event={"ID":"424cb238-c952-43a9-bc11-61549235c627","Type":"ContainerStarted","Data":"0c3b5c155da97f6ffc83447984ec9a9e539050b86bcd7d0b75682e4cbe7563ee"}
Jul 27 17:31:15.787613 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:15.787586    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:15.789559 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:31:15.789528    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c77818_4edf_4248_9e9b_8a8e78246df4.slice/crio-8cdaf18e5a147cefe7029939a38c7d7c9a2a82fe64587d6288ca82aaf464ca98 WatchSource:0}: Error finding container 8cdaf18e5a147cefe7029939a38c7d7c9a2a82fe64587d6288ca82aaf464ca98: Status 404 returned error can't find the container with id 8cdaf18e5a147cefe7029939a38c7d7c9a2a82fe64587d6288ca82aaf464ca98
Jul 27 17:31:16.665735 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:16.665689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" event={"ID":"00c77818-4edf-4248-9e9b-8a8e78246df4","Type":"ContainerStarted","Data":"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"}
Jul 27 17:31:16.666121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:16.665757    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:16.666121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:16.665779    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" event={"ID":"00c77818-4edf-4248-9e9b-8a8e78246df4","Type":"ContainerStarted","Data":"8cdaf18e5a147cefe7029939a38c7d7c9a2a82fe64587d6288ca82aaf464ca98"}
Jul 27 17:31:17.540771 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:17.540721    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" podStartSLOduration=2.54070438 podStartE2EDuration="2.54070438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:16.685033139 +0000 UTC m=+2096.517398115" watchObservedRunningTime="2026-07-27 17:31:17.54070438 +0000 UTC m=+2097.373069356"
Jul 27 17:31:17.542027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:17.542009    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:31:18.676565 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:18.676525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" event={"ID":"424cb238-c952-43a9-bc11-61549235c627","Type":"ContainerStarted","Data":"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"}
Jul 27 17:31:18.676949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:18.676602    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:18.702897 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:18.701657    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" podStartSLOduration=2.171564413 podStartE2EDuration="4.701639148s" totalImagesPullingTime="2.530074735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:18.697478931 +0000 UTC m=+2098.529843933" watchObservedRunningTime="2026-07-27 17:31:18.701639148 +0000 UTC m=+2098.534004129"
Jul 27 17:31:20.724863 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:20.724835    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:31:20.728656 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:20.728634    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:31:24.686028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:24.685995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:27.672984 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:27.672951    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:36.827059 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:36.827027    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:36.827596 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:36.827235    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" podUID="424cb238-c952-43a9-bc11-61549235c627" containerName="maas-api" containerID="cri-o://049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5" gracePeriod=30
Jul 27 17:31:36.903852 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:36.903824    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:36.904105 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:36.904078    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" podUID="00c77818-4edf-4248-9e9b-8a8e78246df4" containerName="payload-processing" containerID="cri-o://f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597" gracePeriod=30
Jul 27 17:31:37.138571 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.138546    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:37.166737 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.166710    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:37.196912 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.196856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-587cj\" (UniqueName: \"kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj\") pod \"424cb238-c952-43a9-bc11-61549235c627\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") "
Jul 27 17:31:37.197026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.196966    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") pod \"424cb238-c952-43a9-bc11-61549235c627\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") "
Jul 27 17:31:37.197026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.196997    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume\") pod \"00c77818-4edf-4248-9e9b-8a8e78246df4\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") "
Jul 27 17:31:37.197103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.197052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b46vm\" (UniqueName: \"kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm\") pod \"00c77818-4edf-4248-9e9b-8a8e78246df4\" (UID: \"00c77818-4edf-4248-9e9b-8a8e78246df4\") "
Jul 27 17:31:37.197103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.197098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca\") pod \"424cb238-c952-43a9-bc11-61549235c627\" (UID: \"424cb238-c952-43a9-bc11-61549235c627\") "
Jul 27 17:31:37.197447 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.197418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00c77818-4edf-4248-9e9b-8a8e78246df4" (UID: "00c77818-4edf-4248-9e9b-8a8e78246df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:37.197575 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.197499    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "424cb238-c952-43a9-bc11-61549235c627" (UID: "424cb238-c952-43a9-bc11-61549235c627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:37.198978 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.198946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj" (OuterVolumeSpecName: "kube-api-access-587cj") pod "424cb238-c952-43a9-bc11-61549235c627" (UID: "424cb238-c952-43a9-bc11-61549235c627"). InnerVolumeSpecName "kube-api-access-587cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:37.199065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.199010    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm" (OuterVolumeSpecName: "kube-api-access-b46vm") pod "00c77818-4edf-4248-9e9b-8a8e78246df4" (UID: "00c77818-4edf-4248-9e9b-8a8e78246df4"). InnerVolumeSpecName "kube-api-access-b46vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:37.199065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.199026    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "424cb238-c952-43a9-bc11-61549235c627" (UID: "424cb238-c952-43a9-bc11-61549235c627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:37.297958 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.297933    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-587cj\" (UniqueName: \"kubernetes.io/projected/424cb238-c952-43a9-bc11-61549235c627-kube-api-access-587cj\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:31:37.297958 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.297954    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cb238-c952-43a9-bc11-61549235c627-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:31:37.298097 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.297964    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c77818-4edf-4248-9e9b-8a8e78246df4-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:31:37.298097 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.297972    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b46vm\" (UniqueName: \"kubernetes.io/projected/00c77818-4edf-4248-9e9b-8a8e78246df4-kube-api-access-b46vm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:31:37.298097 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.297981    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cb238-c952-43a9-bc11-61549235c627-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:31:37.758712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.758684    2564 generic.go:356] "Generic (PLEG): container finished" podID="424cb238-c952-43a9-bc11-61549235c627" containerID="049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5" exitCode=0
Jul 27 17:31:37.758874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.758718    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"
Jul 27 17:31:37.758874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.758770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" event={"ID":"424cb238-c952-43a9-bc11-61549235c627","Type":"ContainerDied","Data":"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"}
Jul 27 17:31:37.758874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.758816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx" event={"ID":"424cb238-c952-43a9-bc11-61549235c627","Type":"ContainerDied","Data":"0c3b5c155da97f6ffc83447984ec9a9e539050b86bcd7d0b75682e4cbe7563ee"}
Jul 27 17:31:37.758874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.758844    2564 scope.go:117] "RemoveContainer" containerID="049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"
Jul 27 17:31:37.760163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.760134    2564 generic.go:356] "Generic (PLEG): container finished" podID="00c77818-4edf-4248-9e9b-8a8e78246df4" containerID="f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597" exitCode=0
Jul 27 17:31:37.760284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.760170    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"
Jul 27 17:31:37.760284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.760168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" event={"ID":"00c77818-4edf-4248-9e9b-8a8e78246df4","Type":"ContainerDied","Data":"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"}
Jul 27 17:31:37.760284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.760264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn" event={"ID":"00c77818-4edf-4248-9e9b-8a8e78246df4","Type":"ContainerDied","Data":"8cdaf18e5a147cefe7029939a38c7d7c9a2a82fe64587d6288ca82aaf464ca98"}
Jul 27 17:31:37.768398 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.768382    2564 scope.go:117] "RemoveContainer" containerID="049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"
Jul 27 17:31:37.768658 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:31:37.768640    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5\": container with ID starting with 049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5 not found: ID does not exist" containerID="049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"
Jul 27 17:31:37.768723 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.768663    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5"} err="failed to get container status \"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5\": rpc error: code = NotFound desc = could not find container \"049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5\": container with ID starting with 049f2a2f6924edcd73fae2b0fed1f94e9c4fb7bb6014d8370c54964f4469f8b5 not found: ID does not exist"
Jul 27 17:31:37.768723 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.768679    2564 scope.go:117] "RemoveContainer" containerID="f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"
Jul 27 17:31:37.776130 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.776111    2564 scope.go:117] "RemoveContainer" containerID="f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"
Jul 27 17:31:37.776370 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:31:37.776352    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597\": container with ID starting with f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597 not found: ID does not exist" containerID="f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"
Jul 27 17:31:37.776423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.776378    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597"} err="failed to get container status \"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597\": rpc error: code = NotFound desc = could not find container \"f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597\": container with ID starting with f0fe6586e77d46a00eab256ef624b128521428d4e8689ac8221b1e4f64ab2597 not found: ID does not exist"
Jul 27 17:31:37.783808 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.783787    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:37.788999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.788980    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18117344-59b585cd59-kjmjx"]
Jul 27 17:31:37.802054 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.802026    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:37.804341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:37.804324    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18117344-5bc7979646-cvtxn"]
Jul 27 17:31:38.658754 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:38.658721    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00c77818-4edf-4248-9e9b-8a8e78246df4" path="/var/lib/kubelet/pods/00c77818-4edf-4248-9e9b-8a8e78246df4/volumes"
Jul 27 17:31:38.659141 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:38.659118    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="424cb238-c952-43a9-bc11-61549235c627" path="/var/lib/kubelet/pods/424cb238-c952-43a9-bc11-61549235c627/volumes"
Jul 27 17:31:39.584814 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:31:39.584779    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:32:12.316378 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316340    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:32:12.316865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316825    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00c77818-4edf-4248-9e9b-8a8e78246df4" containerName="payload-processing"
Jul 27 17:32:12.316865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316839    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="00c77818-4edf-4248-9e9b-8a8e78246df4" containerName="payload-processing"
Jul 27 17:32:12.316974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316868    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="424cb238-c952-43a9-bc11-61549235c627" containerName="maas-api"
Jul 27 17:32:12.316974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316894    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="424cb238-c952-43a9-bc11-61549235c627" containerName="maas-api"
Jul 27 17:32:12.316974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316971    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="00c77818-4edf-4248-9e9b-8a8e78246df4" containerName="payload-processing"
Jul 27 17:32:12.317083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.316987    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="424cb238-c952-43a9-bc11-61549235c627" containerName="maas-api"
Jul 27 17:32:12.320165 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.320147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.322782 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.322759    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-736d5a21-serving-cert\""
Jul 27 17:32:12.322890 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.322862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:32:12.324243 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.324190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:32:12.324243 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.324190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:32:12.331929 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.331907    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:32:12.469177 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.469137    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.469335 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.469195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.469335 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.469284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqgg\" (UniqueName: \"kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.569788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.569708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.569788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.569763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.569964 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.569798    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqgg\" (UniqueName: \"kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.569964 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:32:12.569853    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-serving-cert: secret "maas-api-e2e-mt-736d5a21-serving-cert" not found
Jul 27 17:32:12.569964 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:32:12.569947    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls podName:f3200523-91f3-4140-9ad6-a801bad4e288 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:13.069928285 +0000 UTC m=+2152.902293240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls") pod "maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" (UID: "f3200523-91f3-4140-9ad6-a801bad4e288") : secret "maas-api-e2e-mt-736d5a21-serving-cert" not found
Jul 27 17:32:12.570418 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.570399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:12.580907 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:12.580868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqgg\" (UniqueName: \"kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:13.073615 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.073586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:13.076026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.075998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") pod \"maas-api-e2e-mt-736d5a21-769f6d4777-sspdt\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:13.226081 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.226046    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:32:13.229642 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.229623    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.230599 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.230582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:13.232682 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.232655    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-736d5a21\""
Jul 27 17:32:13.232800 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.232720    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-736d5a21-dockercfg-9hs7b\""
Jul 27 17:32:13.242550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.242529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:32:13.296238 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.296184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:32:13.300988 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.300966    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.310985 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.310958    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:32:13.373932 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.373910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:32:13.376058 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:32:13.376026    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3200523_91f3_4140_9ad6_a801bad4e288.slice/crio-e1324f8ab838a6bd5b4d25777bee910d8a82b247851da3112f4f5e0d9dc6ad54 WatchSource:0}: Error finding container e1324f8ab838a6bd5b4d25777bee910d8a82b247851da3112f4f5e0d9dc6ad54: Status 404 returned error can't find the container with id e1324f8ab838a6bd5b4d25777bee910d8a82b247851da3112f4f5e0d9dc6ad54
Jul 27 17:32:13.376310 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.376283    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.376418 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.376399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zpw\" (UniqueName: \"kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.377487 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.377471    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:32:13.477184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.477155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zpw\" (UniqueName: \"kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.477353 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.477188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbn7\" (UniqueName: \"kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.477353 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.477242    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.477353 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.477293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.478129 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.478108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.486943 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.486924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zpw\" (UniqueName: \"kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw\") pod \"payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") " pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.540671 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.540638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.578039 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.577997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.578182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.578148    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbn7\" (UniqueName: \"kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.579008 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.578987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.587647 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.587611    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbn7\" (UniqueName: \"kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7\") pod \"payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.612749 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.612724    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.677091 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.677068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:32:13.679914 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:32:13.679835    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed3080f_c555_4a7a_a997_a271c143c377.slice/crio-b6bb2b712bef0fdb5d394420978fa4702f566b6eb7c91e93c0c6032a88e9cafd WatchSource:0}: Error finding container b6bb2b712bef0fdb5d394420978fa4702f566b6eb7c91e93c0c6032a88e9cafd: Status 404 returned error can't find the container with id b6bb2b712bef0fdb5d394420978fa4702f566b6eb7c91e93c0c6032a88e9cafd
Jul 27 17:32:13.769733 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.769710    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:32:13.772279 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:32:13.772256    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47785044_cc95_4651_b834_01d542cfdbf4.slice/crio-ce446d934c0899b2e0c12e523e4120667a0cc0a56bec5c72a8c829346ac6c1bd WatchSource:0}: Error finding container ce446d934c0899b2e0c12e523e4120667a0cc0a56bec5c72a8c829346ac6c1bd: Status 404 returned error can't find the container with id ce446d934c0899b2e0c12e523e4120667a0cc0a56bec5c72a8c829346ac6c1bd
Jul 27 17:32:13.913639 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.913603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" event={"ID":"f3200523-91f3-4140-9ad6-a801bad4e288","Type":"ContainerStarted","Data":"e1324f8ab838a6bd5b4d25777bee910d8a82b247851da3112f4f5e0d9dc6ad54"}
Jul 27 17:32:13.915059 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.915021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" event={"ID":"47785044-cc95-4651-b834-01d542cfdbf4","Type":"ContainerStarted","Data":"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"}
Jul 27 17:32:13.915231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.915167    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:13.915231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.915190    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" event={"ID":"47785044-cc95-4651-b834-01d542cfdbf4","Type":"ContainerStarted","Data":"ce446d934c0899b2e0c12e523e4120667a0cc0a56bec5c72a8c829346ac6c1bd"}
Jul 27 17:32:13.916409 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.916386    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" event={"ID":"fed3080f-c555-4a7a-a997-a271c143c377","Type":"ContainerStarted","Data":"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"}
Jul 27 17:32:13.916503 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.916425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" event={"ID":"fed3080f-c555-4a7a-a997-a271c143c377","Type":"ContainerStarted","Data":"b6bb2b712bef0fdb5d394420978fa4702f566b6eb7c91e93c0c6032a88e9cafd"}
Jul 27 17:32:13.916503 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.916446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:13.944348 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.944295    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" podStartSLOduration=0.944275276 podStartE2EDuration="944.275276ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:13.941688279 +0000 UTC m=+2153.774053276" watchObservedRunningTime="2026-07-27 17:32:13.944275276 +0000 UTC m=+2153.776640255"
Jul 27 17:32:13.979447 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:13.979402    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" podStartSLOduration=0.979386134 podStartE2EDuration="979.386134ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:13.979227408 +0000 UTC m=+2153.811592383" watchObservedRunningTime="2026-07-27 17:32:13.979386134 +0000 UTC m=+2153.811751110"
Jul 27 17:32:15.927280 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:15.927246    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" event={"ID":"f3200523-91f3-4140-9ad6-a801bad4e288","Type":"ContainerStarted","Data":"bd30bc8d70c39e96d76eace99d742b049b6c870c784351fdebac95f19384039a"}
Jul 27 17:32:15.927748 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:15.927356    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:15.948246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:15.948198    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" podStartSLOduration=1.7637264959999999 podStartE2EDuration="3.948184528s" totalImagesPullingTime="2.184458032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:15.946425166 +0000 UTC m=+2155.778790152" watchObservedRunningTime="2026-07-27 17:32:15.948184528 +0000 UTC m=+2155.780549551"
Jul 27 17:32:16.708835 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:16.708806    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:32:21.937627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:21.937597    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:32:24.923712 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:24.923685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:32:24.924098 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:24.923921    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:32:34.786217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.786184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:32:34.794095 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.794063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.798766 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.798741    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-736d5a21\"/\"openshift-service-ca.crt\""
Jul 27 17:32:34.798766 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.798763    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-736d5a21\"/\"default-dockercfg-7cslk\""
Jul 27 17:32:34.799042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.798763    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-736d5a21\"/\"e2e-lifecycle-model-736d5a21-kserve-self-signed-certs\""
Jul 27 17:32:34.799042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.798750    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-736d5a21\"/\"kube-root-ca.crt\""
Jul 27 17:32:34.802443 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.802422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:32:34.839410 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839386    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.839513 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.839513 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839437    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.839513 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnzn\" (UniqueName: \"kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.839513 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.839689 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.839521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940176 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnzn\" (UniqueName: \"kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940524 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940309    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.940739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.940724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.942436 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.942410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.942578 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.942562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:34.949123 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:34.949100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnzn\" (UniqueName: \"kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn\") pod \"e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") " pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:35.104369 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:35.104292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:35.233124 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:35.233102    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:32:35.235088 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:32:35.235060    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b5c53af_530e_47d2_9921_d1f66598b256.slice/crio-7a85cd223b62bf8d9ff4aba2526087c709e240d1d11ef1d74b6a0f35601aa366 WatchSource:0}: Error finding container 7a85cd223b62bf8d9ff4aba2526087c709e240d1d11ef1d74b6a0f35601aa366: Status 404 returned error can't find the container with id 7a85cd223b62bf8d9ff4aba2526087c709e240d1d11ef1d74b6a0f35601aa366
Jul 27 17:32:36.016923 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:36.016868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" event={"ID":"8b5c53af-530e-47d2-9921-d1f66598b256","Type":"ContainerStarted","Data":"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"}
Jul 27 17:32:36.017294 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:36.016933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" event={"ID":"8b5c53af-530e-47d2-9921-d1f66598b256","Type":"ContainerStarted","Data":"7a85cd223b62bf8d9ff4aba2526087c709e240d1d11ef1d74b6a0f35601aa366"}
Jul 27 17:32:36.042729 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:36.042665    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" podStartSLOduration=2.042650998 podStartE2EDuration="2.042650998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:36.040774702 +0000 UTC m=+2175.873139679" watchObservedRunningTime="2026-07-27 17:32:36.042650998 +0000 UTC m=+2175.875015974"
Jul 27 17:32:37.703564 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:37.703530    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:32:45.104986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:45.104949    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:45.105457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:45.104999    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:45.117130 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:45.117105    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:46.072806 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:46.072780    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:32:53.602164 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:32:53.602127    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:33:15.394631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:15.394597    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:33:17.028826 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.028794    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:33:17.029308 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.028999    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" podUID="f3200523-91f3-4140-9ad6-a801bad4e288" containerName="maas-api" containerID="cri-o://bd30bc8d70c39e96d76eace99d742b049b6c870c784351fdebac95f19384039a" gracePeriod=30
Jul 27 17:33:17.130396 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.130367    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:33:17.130618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.130594    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" podUID="fed3080f-c555-4a7a-a997-a271c143c377" containerName="payload-processing" containerID="cri-o://a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627" gracePeriod=30
Jul 27 17:33:17.147582 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.147560    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:33:17.147776 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.147754    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" podUID="47785044-cc95-4651-b834-01d542cfdbf4" containerName="payload-pre-processing" containerID="cri-o://57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d" gracePeriod=30
Jul 27 17:33:17.214993 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.214803    2564 generic.go:356] "Generic (PLEG): container finished" podID="f3200523-91f3-4140-9ad6-a801bad4e288" containerID="bd30bc8d70c39e96d76eace99d742b049b6c870c784351fdebac95f19384039a" exitCode=0
Jul 27 17:33:17.214993 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.214937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" event={"ID":"f3200523-91f3-4140-9ad6-a801bad4e288","Type":"ContainerDied","Data":"bd30bc8d70c39e96d76eace99d742b049b6c870c784351fdebac95f19384039a"}
Jul 27 17:33:17.293972 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.293912    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:33:17.373300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.373277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") pod \"f3200523-91f3-4140-9ad6-a801bad4e288\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") "
Jul 27 17:33:17.373422 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.373357    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlqgg\" (UniqueName: \"kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg\") pod \"f3200523-91f3-4140-9ad6-a801bad4e288\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") "
Jul 27 17:33:17.373422 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.373390    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca\") pod \"f3200523-91f3-4140-9ad6-a801bad4e288\" (UID: \"f3200523-91f3-4140-9ad6-a801bad4e288\") "
Jul 27 17:33:17.373780 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.373755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3200523-91f3-4140-9ad6-a801bad4e288" (UID: "f3200523-91f3-4140-9ad6-a801bad4e288"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:17.375456 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.375431    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3200523-91f3-4140-9ad6-a801bad4e288" (UID: "f3200523-91f3-4140-9ad6-a801bad4e288"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:17.375892 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.375853    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg" (OuterVolumeSpecName: "kube-api-access-zlqgg") pod "f3200523-91f3-4140-9ad6-a801bad4e288" (UID: "f3200523-91f3-4140-9ad6-a801bad4e288"). InnerVolumeSpecName "kube-api-access-zlqgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:17.414276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.414256    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:33:17.420464 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.420445    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:33:17.474266 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.474242    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3200523-91f3-4140-9ad6-a801bad4e288-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.474266 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.474264    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3200523-91f3-4140-9ad6-a801bad4e288-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.474397 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.474273    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlqgg\" (UniqueName: \"kubernetes.io/projected/f3200523-91f3-4140-9ad6-a801bad4e288-kube-api-access-zlqgg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.575217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575150    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbbn7\" (UniqueName: \"kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7\") pod \"47785044-cc95-4651-b834-01d542cfdbf4\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") "
Jul 27 17:33:17.575217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575203    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume\") pod \"fed3080f-c555-4a7a-a997-a271c143c377\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") "
Jul 27 17:33:17.575217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575227    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8zpw\" (UniqueName: \"kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw\") pod \"fed3080f-c555-4a7a-a997-a271c143c377\" (UID: \"fed3080f-c555-4a7a-a997-a271c143c377\") "
Jul 27 17:33:17.575462 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume\") pod \"47785044-cc95-4651-b834-01d542cfdbf4\" (UID: \"47785044-cc95-4651-b834-01d542cfdbf4\") "
Jul 27 17:33:17.575636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575600    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fed3080f-c555-4a7a-a997-a271c143c377" (UID: "fed3080f-c555-4a7a-a997-a271c143c377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:17.575746 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.575651    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47785044-cc95-4651-b834-01d542cfdbf4" (UID: "47785044-cc95-4651-b834-01d542cfdbf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:17.577264 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.577241    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw" (OuterVolumeSpecName: "kube-api-access-r8zpw") pod "fed3080f-c555-4a7a-a997-a271c143c377" (UID: "fed3080f-c555-4a7a-a997-a271c143c377"). InnerVolumeSpecName "kube-api-access-r8zpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:17.577545 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.577528    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7" (OuterVolumeSpecName: "kube-api-access-tbbn7") pod "47785044-cc95-4651-b834-01d542cfdbf4" (UID: "47785044-cc95-4651-b834-01d542cfdbf4"). InnerVolumeSpecName "kube-api-access-tbbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:17.675999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.675971    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8zpw\" (UniqueName: \"kubernetes.io/projected/fed3080f-c555-4a7a-a997-a271c143c377-kube-api-access-r8zpw\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.675999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.675993    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47785044-cc95-4651-b834-01d542cfdbf4-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.675999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.676002    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbn7\" (UniqueName: \"kubernetes.io/projected/47785044-cc95-4651-b834-01d542cfdbf4-kube-api-access-tbbn7\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:17.676194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:17.676011    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fed3080f-c555-4a7a-a997-a271c143c377-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:18.221467 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.221431    2564 generic.go:356] "Generic (PLEG): container finished" podID="47785044-cc95-4651-b834-01d542cfdbf4" containerID="57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d" exitCode=0
Jul 27 17:33:18.221911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.221475    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"
Jul 27 17:33:18.221911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.221471    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" event={"ID":"47785044-cc95-4651-b834-01d542cfdbf4","Type":"ContainerDied","Data":"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"}
Jul 27 17:33:18.221911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.221589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn" event={"ID":"47785044-cc95-4651-b834-01d542cfdbf4","Type":"ContainerDied","Data":"ce446d934c0899b2e0c12e523e4120667a0cc0a56bec5c72a8c829346ac6c1bd"}
Jul 27 17:33:18.221911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.221612    2564 scope.go:117] "RemoveContainer" containerID="57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"
Jul 27 17:33:18.223075 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.222917    2564 generic.go:356] "Generic (PLEG): container finished" podID="fed3080f-c555-4a7a-a997-a271c143c377" containerID="a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627" exitCode=0
Jul 27 17:33:18.223075 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.222954    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"
Jul 27 17:33:18.223075 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.222996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" event={"ID":"fed3080f-c555-4a7a-a997-a271c143c377","Type":"ContainerDied","Data":"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"}
Jul 27 17:33:18.223075 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.223045    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf" event={"ID":"fed3080f-c555-4a7a-a997-a271c143c377","Type":"ContainerDied","Data":"b6bb2b712bef0fdb5d394420978fa4702f566b6eb7c91e93c0c6032a88e9cafd"}
Jul 27 17:33:18.224518 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.224494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt" event={"ID":"f3200523-91f3-4140-9ad6-a801bad4e288","Type":"ContainerDied","Data":"e1324f8ab838a6bd5b4d25777bee910d8a82b247851da3112f4f5e0d9dc6ad54"}
Jul 27 17:33:18.224518 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.224523    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"
Jul 27 17:33:18.232367 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.232337    2564 scope.go:117] "RemoveContainer" containerID="57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"
Jul 27 17:33:18.239407 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:33:18.239347    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d\": container with ID starting with 57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d not found: ID does not exist" containerID="57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"
Jul 27 17:33:18.239407 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.239388    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d"} err="failed to get container status \"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d\": rpc error: code = NotFound desc = could not find container \"57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d\": container with ID starting with 57b52aa6acf2fdfceed0cd5752a26ea93f7e09eb96e5e3431b31ad64b27bf56d not found: ID does not exist"
Jul 27 17:33:18.240129 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.239411    2564 scope.go:117] "RemoveContainer" containerID="a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"
Jul 27 17:33:18.251170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.251150    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:33:18.251170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.251167    2564 scope.go:117] "RemoveContainer" containerID="a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"
Jul 27 17:33:18.251468 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:33:18.251450    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627\": container with ID starting with a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627 not found: ID does not exist" containerID="a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"
Jul 27 17:33:18.251541 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.251478    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627"} err="failed to get container status \"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627\": rpc error: code = NotFound desc = could not find container \"a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627\": container with ID starting with a525973b08b05dd537016b48ee2d9aa7121bab60a4d6982ab51c3c0d69741627 not found: ID does not exist"
Jul 27 17:33:18.251541 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.251515    2564 scope.go:117] "RemoveContainer" containerID="bd30bc8d70c39e96d76eace99d742b049b6c870c784351fdebac95f19384039a"
Jul 27 17:33:18.257245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.257226    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-736d5a21-5b98d9fdd8-b7czn"]
Jul 27 17:33:18.268457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.268436    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:33:18.272561 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.272541    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-736d5a21-769f6d4777-sspdt"]
Jul 27 17:33:18.285198 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.285175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:33:18.290769 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.290750    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-736d5a21-5b4d478f94-qcmsf"]
Jul 27 17:33:18.658971 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.658944    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47785044-cc95-4651-b834-01d542cfdbf4" path="/var/lib/kubelet/pods/47785044-cc95-4651-b834-01d542cfdbf4/volumes"
Jul 27 17:33:18.659307 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.659295    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3200523-91f3-4140-9ad6-a801bad4e288" path="/var/lib/kubelet/pods/f3200523-91f3-4140-9ad6-a801bad4e288/volumes"
Jul 27 17:33:18.659608 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:18.659596    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed3080f-c555-4a7a-a997-a271c143c377" path="/var/lib/kubelet/pods/fed3080f-c555-4a7a-a997-a271c143c377/volumes"
Jul 27 17:33:26.091777 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:26.091749    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:33:29.427068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:29.427032    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:33:44.446090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.446038    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" podUID="8b5c53af-530e-47d2-9921-d1f66598b256" containerName="main" containerID="cri-o://a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae" gracePeriod=45
Jul 27 17:33:44.696618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.696563    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:33:44.892104 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892164    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892181    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892216    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxnzn\" (UniqueName: \"kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892473 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892257    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs\") pod \"8b5c53af-530e-47d2-9921-d1f66598b256\" (UID: \"8b5c53af-530e-47d2-9921-d1f66598b256\") "
Jul 27 17:33:44.892473 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892303    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:44.892473 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892420    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:44.892635 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892467    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home" (OuterVolumeSpecName: "home") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:44.892635 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.892485    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:44.894287 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.894263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm" (OuterVolumeSpecName: "dshm") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:44.894287 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.894279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:44.894450 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.894331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn" (OuterVolumeSpecName: "kube-api-access-hxnzn") pod "8b5c53af-530e-47d2-9921-d1f66598b256" (UID: "8b5c53af-530e-47d2-9921-d1f66598b256"). InnerVolumeSpecName "kube-api-access-hxnzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:44.993033 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.992977    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:44.993033 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.992997    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:44.993033 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.993007    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b5c53af-530e-47d2-9921-d1f66598b256-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:44.993033 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.993016    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnzn\" (UniqueName: \"kubernetes.io/projected/8b5c53af-530e-47d2-9921-d1f66598b256-kube-api-access-hxnzn\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:44.993033 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:44.993024    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b5c53af-530e-47d2-9921-d1f66598b256-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:33:45.350641 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.350559    2564 generic.go:356] "Generic (PLEG): container finished" podID="8b5c53af-530e-47d2-9921-d1f66598b256" containerID="a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae" exitCode=0
Jul 27 17:33:45.350641 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.350609    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"
Jul 27 17:33:45.350839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.350642    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" event={"ID":"8b5c53af-530e-47d2-9921-d1f66598b256","Type":"ContainerDied","Data":"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"}
Jul 27 17:33:45.350839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.350689    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8" event={"ID":"8b5c53af-530e-47d2-9921-d1f66598b256","Type":"ContainerDied","Data":"7a85cd223b62bf8d9ff4aba2526087c709e240d1d11ef1d74b6a0f35601aa366"}
Jul 27 17:33:45.350839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.350708    2564 scope.go:117] "RemoveContainer" containerID="a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"
Jul 27 17:33:45.361323 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.361304    2564 scope.go:117] "RemoveContainer" containerID="a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"
Jul 27 17:33:45.361769 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:33:45.361710    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae\": container with ID starting with a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae not found: ID does not exist" containerID="a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"
Jul 27 17:33:45.361769 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.361743    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae"} err="failed to get container status \"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae\": rpc error: code = NotFound desc = could not find container \"a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae\": container with ID starting with a7b9d23f8742496d451771355c0520f64d5173a24f7c2f1eed80708c51eb07ae not found: ID does not exist"
Jul 27 17:33:45.374149 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.374124    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:33:45.377465 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:45.377422    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-736d5a21/e2e-lifecycle-model-736d5a21-kserve-64cc79568c-crrz8"]
Jul 27 17:33:46.658061 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:33:46.658024    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5c53af-530e-47d2-9921-d1f66598b256" path="/var/lib/kubelet/pods/8b5c53af-530e-47d2-9921-d1f66598b256/volumes"
Jul 27 17:34:22.902467 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:34:22.902432    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:34:35.099634 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:34:35.099595    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:35:20.186848 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.186817    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187156    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed3080f-c555-4a7a-a997-a271c143c377" containerName="payload-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187167    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed3080f-c555-4a7a-a997-a271c143c377" containerName="payload-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187175    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47785044-cc95-4651-b834-01d542cfdbf4" containerName="payload-pre-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187180    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="47785044-cc95-4651-b834-01d542cfdbf4" containerName="payload-pre-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187196    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3200523-91f3-4140-9ad6-a801bad4e288" containerName="maas-api"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187201    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3200523-91f3-4140-9ad6-a801bad4e288" containerName="maas-api"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187211    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5c53af-530e-47d2-9921-d1f66598b256" containerName="main"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187216    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5c53af-530e-47d2-9921-d1f66598b256" containerName="main"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187260    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="47785044-cc95-4651-b834-01d542cfdbf4" containerName="payload-pre-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187267    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed3080f-c555-4a7a-a997-a271c143c377" containerName="payload-processing"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187276    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5c53af-530e-47d2-9921-d1f66598b256" containerName="main"
Jul 27 17:35:20.187349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.187283    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3200523-91f3-4140-9ad6-a801bad4e288" containerName="maas-api"
Jul 27 17:35:20.190103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.190087    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.193184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.193146    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a6f598-serving-cert\""
Jul 27 17:35:20.193184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.193169    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:35:20.194647 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.194625    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:35:20.194751 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.194670    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:35:20.204197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.204175    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:35:20.382562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.382529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6cv\" (UniqueName: \"kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.382711 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.382588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.382711 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.382653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.483559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.483493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.483559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.483525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.483739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.483562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6cv\" (UniqueName: \"kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.484215 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.484184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.485845 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.485825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.492314 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.492297    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6cv\" (UniqueName: \"kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv\") pod \"maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.500060 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.500041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:20.636341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.636316    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:35:20.638851 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:20.638822    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1f583d_681b_42b8_a346_119157b07d0f.slice/crio-ddd02f89df843e14ed8c3ea1554a91461e2b84148481714c535a30ffadf637a1 WatchSource:0}: Error finding container ddd02f89df843e14ed8c3ea1554a91461e2b84148481714c535a30ffadf637a1: Status 404 returned error can't find the container with id ddd02f89df843e14ed8c3ea1554a91461e2b84148481714c535a30ffadf637a1
Jul 27 17:35:20.750554 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:20.750486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" event={"ID":"3c1f583d-681b-42b8-a346-119157b07d0f","Type":"ContainerStarted","Data":"ddd02f89df843e14ed8c3ea1554a91461e2b84148481714c535a30ffadf637a1"}
Jul 27 17:35:21.031397 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.031321    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:35:21.035669 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.035653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.038818 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.038788    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a6f598-dockercfg-7dscl\""
Jul 27 17:35:21.038975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.038889    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a6f598\""
Jul 27 17:35:21.044555 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.044532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:35:21.072202 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.072180    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:35:21.075265 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.075251    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.086630 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.086611    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:35:21.188910 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.188867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.189221 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.188959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6ln\" (UniqueName: \"kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.189221 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.189006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6zx\" (UniqueName: \"kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.189221 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.189091    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.289928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.289851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.289928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.289905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.290104 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.289966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6ln\" (UniqueName: \"kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.290104 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.290009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6zx\" (UniqueName: \"kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.290634 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.290615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.290688 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.290620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.299789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.299765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6ln\" (UniqueName: \"kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln\") pod \"payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.299921 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.299827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6zx\" (UniqueName: \"kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx\") pod \"payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.346604 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.346576    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:21.385089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.384906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:21.712478 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.712456    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:35:21.713658 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:21.713637    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2842ec04_fe56_46ce_9b19_bdf710638d7f.slice/crio-c0727d58201650ee413a4b86647113853d96859d4fd9cb9c1912e45ba64f3ef3 WatchSource:0}: Error finding container c0727d58201650ee413a4b86647113853d96859d4fd9cb9c1912e45ba64f3ef3: Status 404 returned error can't find the container with id c0727d58201650ee413a4b86647113853d96859d4fd9cb9c1912e45ba64f3ef3
Jul 27 17:35:21.755573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.755545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" event={"ID":"2842ec04-fe56-46ce-9b19-bdf710638d7f","Type":"ContainerStarted","Data":"c0727d58201650ee413a4b86647113853d96859d4fd9cb9c1912e45ba64f3ef3"}
Jul 27 17:35:21.756925 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.756902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" event={"ID":"3c1f583d-681b-42b8-a346-119157b07d0f","Type":"ContainerStarted","Data":"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"}
Jul 27 17:35:21.757021 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.757003    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:21.773175 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.773125    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:35:21.773937 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:21.773914    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd9b054_40f4_4c9a_8ba5_e98d0b24744f.slice/crio-05861d266de903630a0a8526253001ea7a3c19a1dd9e269769254c3aaed10cdd WatchSource:0}: Error finding container 05861d266de903630a0a8526253001ea7a3c19a1dd9e269769254c3aaed10cdd: Status 404 returned error can't find the container with id 05861d266de903630a0a8526253001ea7a3c19a1dd9e269769254c3aaed10cdd
Jul 27 17:35:21.801002 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:21.800954    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" podStartSLOduration=1.303647658 podStartE2EDuration="1.800938888s" totalImagesPullingTime="497.29123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:21.799051104 +0000 UTC m=+2341.631416080" watchObservedRunningTime="2026-07-27 17:35:21.800938888 +0000 UTC m=+2341.633303988"
Jul 27 17:35:22.763007 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.762975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" event={"ID":"2842ec04-fe56-46ce-9b19-bdf710638d7f","Type":"ContainerStarted","Data":"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"}
Jul 27 17:35:22.763443 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.763029    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:22.764622 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.764597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" event={"ID":"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f","Type":"ContainerStarted","Data":"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"}
Jul 27 17:35:22.764747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.764638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" event={"ID":"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f","Type":"ContainerStarted","Data":"05861d266de903630a0a8526253001ea7a3c19a1dd9e269769254c3aaed10cdd"}
Jul 27 17:35:22.764800 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.764763    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:22.782361 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.782318    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" podStartSLOduration=1.78230529 podStartE2EDuration="1.78230529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:22.780978113 +0000 UTC m=+2342.613343131" watchObservedRunningTime="2026-07-27 17:35:22.78230529 +0000 UTC m=+2342.614670245"
Jul 27 17:35:22.799372 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:22.799339    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" podStartSLOduration=1.799326208 podStartE2EDuration="1.799326208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:22.798217541 +0000 UTC m=+2342.630582518" watchObservedRunningTime="2026-07-27 17:35:22.799326208 +0000 UTC m=+2342.631691216"
Jul 27 17:35:24.678667 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:24.678637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:35:27.770691 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:27.770662    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:35:28.164780 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.164749    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:35:28.168226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.168206    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.170956 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.170935    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7a55cb-serving-cert\""
Jul 27 17:35:28.179242 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.179216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:35:28.240222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.240199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.240331 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.240246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.240331 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.240276    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5zlr\" (UniqueName: \"kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.341567 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.341534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.341721 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.341579    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.341721 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.341604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5zlr\" (UniqueName: \"kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.341721 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:35:28.341704    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-serving-cert: secret "maas-api-e2e-inf-b-7a55cb-serving-cert" not found
Jul 27 17:35:28.341912 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:35:28.341787    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls podName:425cabe6-cb2e-4e34-8c2f-ad44f28de6ae nodeName:}" failed. No retries permitted until 2026-07-27 17:35:28.841765724 +0000 UTC m=+2348.674130693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls") pod "maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" (UID: "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae") : secret "maas-api-e2e-inf-b-7a55cb-serving-cert" not found
Jul 27 17:35:28.342207 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.342188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.354246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.354224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5zlr\" (UniqueName: \"kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.846896 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.846856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:28.849122 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:28.849100    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") pod \"maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:29.080526 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.080490    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:29.104605 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.104544    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:35:29.108121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.108093    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.111107 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.111076    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7a55cb\""
Jul 27 17:35:29.111107 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.111090    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7a55cb-dockercfg-tlwzk\""
Jul 27 17:35:29.117227 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.117206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:35:29.150475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.150313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpx6g\" (UniqueName: \"kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.150475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.150406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.216504 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.216482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:35:29.218435 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:29.218408    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425cabe6_cb2e_4e34_8c2f_ad44f28de6ae.slice/crio-2bd3c747007397447c55532545036fd9b4b470050f95b67f1bddc97e20aacd58 WatchSource:0}: Error finding container 2bd3c747007397447c55532545036fd9b4b470050f95b67f1bddc97e20aacd58: Status 404 returned error can't find the container with id 2bd3c747007397447c55532545036fd9b4b470050f95b67f1bddc97e20aacd58
Jul 27 17:35:29.251717 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.251696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpx6g\" (UniqueName: \"kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.251835 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.251745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.252389 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.252371    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.259915 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.259899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpx6g\" (UniqueName: \"kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g\") pod \"payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.420280 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.420260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.558909 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.556581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:35:29.565149 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:29.565123    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e792b4_b025_44b0_8efa_32a23ede578d.slice/crio-7f0461ac51478b45e478176b9af9f8271d764ba798958675aab03451faf06ed7 WatchSource:0}: Error finding container 7f0461ac51478b45e478176b9af9f8271d764ba798958675aab03451faf06ed7: Status 404 returned error can't find the container with id 7f0461ac51478b45e478176b9af9f8271d764ba798958675aab03451faf06ed7
Jul 27 17:35:29.797860 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.797754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" event={"ID":"a8e792b4-b025-44b0-8efa-32a23ede578d","Type":"ContainerStarted","Data":"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"}
Jul 27 17:35:29.797860 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.797819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" event={"ID":"a8e792b4-b025-44b0-8efa-32a23ede578d","Type":"ContainerStarted","Data":"7f0461ac51478b45e478176b9af9f8271d764ba798958675aab03451faf06ed7"}
Jul 27 17:35:29.797860 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.797849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:29.799147 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.799120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" event={"ID":"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae","Type":"ContainerStarted","Data":"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"}
Jul 27 17:35:29.799242 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.799169    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" event={"ID":"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae","Type":"ContainerStarted","Data":"2bd3c747007397447c55532545036fd9b4b470050f95b67f1bddc97e20aacd58"}
Jul 27 17:35:29.799242 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.799220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:29.819161 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.819118    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" podStartSLOduration=0.819104402 podStartE2EDuration="819.104402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:29.817867805 +0000 UTC m=+2349.650232782" watchObservedRunningTime="2026-07-27 17:35:29.819104402 +0000 UTC m=+2349.651469377"
Jul 27 17:35:29.838469 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:29.838419    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" podStartSLOduration=1.532624034 podStartE2EDuration="1.838398671s" totalImagesPullingTime="305.774637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:29.837065149 +0000 UTC m=+2349.669430125" watchObservedRunningTime="2026-07-27 17:35:29.838398671 +0000 UTC m=+2349.670763646"
Jul 27 17:35:33.770783 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:33.770755    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:35:33.771152 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:33.770824    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:35:35.809636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:35.809608    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:35:37.838722 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.838693    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:35:37.842234 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.842217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.845292 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.845272    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a6f598\"/\"openshift-service-ca.crt\""
Jul 27 17:35:37.846762 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.846744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a6f598\"/\"kube-root-ca.crt\""
Jul 27 17:35:37.846924 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.846819    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a6f598\"/\"default-dockercfg-2vrt6\""
Jul 27 17:35:37.847018 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.846983    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a6f598\"/\"test-model-a6f598-kserve-self-signed-certs\""
Jul 27 17:35:37.855454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.855429    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:35:37.923054 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.923155 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9cm\" (UniqueName: \"kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.923209 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923165    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.923209 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.923294 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923217    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:37.923294 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:37.923271    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.023723 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.023839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.023839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9cm\" (UniqueName: \"kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.023839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.023839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.024088 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.023843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.024156 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.024136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.024248 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.024229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.024307 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.024264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.026093 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.026064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.026226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.026208    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.032853 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.032832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9cm\" (UniqueName: \"kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm\") pod \"test-model-a6f598-kserve-5b49b56b6c-mhztl\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") " pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.151480 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.151459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:38.213136 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.213111    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:35:38.293435 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.293407    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:35:38.295286 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:38.295261    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77c17407_20f5_4a71_b1cf_6ea8e5de8201.slice/crio-0cf335b8833065254881f48aaa79508636fadd051faae4a4dbf94233d74c44cc WatchSource:0}: Error finding container 0cf335b8833065254881f48aaa79508636fadd051faae4a4dbf94233d74c44cc: Status 404 returned error can't find the container with id 0cf335b8833065254881f48aaa79508636fadd051faae4a4dbf94233d74c44cc
Jul 27 17:35:38.842382 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.842346    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" event={"ID":"77c17407-20f5-4a71-b1cf-6ea8e5de8201","Type":"ContainerStarted","Data":"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"}
Jul 27 17:35:38.842382 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.842381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" event={"ID":"77c17407-20f5-4a71-b1cf-6ea8e5de8201","Type":"ContainerStarted","Data":"0cf335b8833065254881f48aaa79508636fadd051faae4a4dbf94233d74c44cc"}
Jul 27 17:35:38.863826 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:38.863784    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" podStartSLOduration=1.863770181 podStartE2EDuration="1.863770181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:38.862132524 +0000 UTC m=+2358.694497501" watchObservedRunningTime="2026-07-27 17:35:38.863770181 +0000 UTC m=+2358.696135156"
Jul 27 17:35:40.806591 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:40.806557    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:35:48.151792 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:48.151751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:48.152190 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:48.151805    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:48.164230 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:48.164204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:48.900685 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:48.900657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:35:49.078707 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:49.078676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:35:56.713435 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.713389    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:35:56.717611 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.717590    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.721760 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.721734    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7a55cb\"/\"test-model-7a55cb-kserve-self-signed-certs\""
Jul 27 17:35:56.722111 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.721791    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7a55cb\"/\"kube-root-ca.crt\""
Jul 27 17:35:56.722111 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.721856    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7a55cb\"/\"openshift-service-ca.crt\""
Jul 27 17:35:56.722284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.722239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7a55cb\"/\"default-dockercfg-gkgvv\""
Jul 27 17:35:56.729471 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.729451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:35:56.865809 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.865975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865819    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.865975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.865975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s964j\" (UniqueName: \"kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.865975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.866138 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.865978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967165 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967165 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967135    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967165 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967157    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967172    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s964j\" (UniqueName: \"kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967393 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967548 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.967597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.967583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.969453 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.969425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.969692 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.969678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:56.976252 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:56.976226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s964j\" (UniqueName: \"kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j\") pod \"test-model-7a55cb-kserve-57c5999fff-bbnh2\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") " pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:57.030634 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:57.030601    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:35:57.161320 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:57.161296    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:35:57.162699 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:35:57.162676    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod590296c0_81af_4102_9dba_2b8aa692fcc7.slice/crio-88c2bc7aacfee923e486336f14ba2d3bc76c1ea5c4b906f1ff1a398cee73a997 WatchSource:0}: Error finding container 88c2bc7aacfee923e486336f14ba2d3bc76c1ea5c4b906f1ff1a398cee73a997: Status 404 returned error can't find the container with id 88c2bc7aacfee923e486336f14ba2d3bc76c1ea5c4b906f1ff1a398cee73a997
Jul 27 17:35:57.931811 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:57.931774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" event={"ID":"590296c0-81af-4102-9dba-2b8aa692fcc7","Type":"ContainerStarted","Data":"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"}
Jul 27 17:35:57.932211 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:57.931819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" event={"ID":"590296c0-81af-4102-9dba-2b8aa692fcc7","Type":"ContainerStarted","Data":"88c2bc7aacfee923e486336f14ba2d3bc76c1ea5c4b906f1ff1a398cee73a997"}
Jul 27 17:35:57.952521 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:35:57.952470    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" podStartSLOduration=1.952454661 podStartE2EDuration="1.952454661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:57.950870617 +0000 UTC m=+2377.783235600" watchObservedRunningTime="2026-07-27 17:35:57.952454661 +0000 UTC m=+2377.784819636"
Jul 27 17:36:03.083064 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:03.083029    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:36:07.031350 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:07.031316    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:07.031742 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:07.031362    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:07.043869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:07.043834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:07.987627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:07.987595    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:20.751867 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:20.751842    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:36:20.756967 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:20.756941    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:36:36.553553 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:36.553524    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:36:37.499426 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:37.499397    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:36:43.083532 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:43.083456    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:36:48.331772 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.331736    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:36:48.332281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.331976    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" podUID="3c1f583d-681b-42b8-a346-119157b07d0f" containerName="maas-api" containerID="cri-o://4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830" gracePeriod=30
Jul 27 17:36:48.402983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.402958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:36:48.403187 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.403162    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" podUID="2842ec04-fe56-46ce-9b19-bdf710638d7f" containerName="payload-processing" containerID="cri-o://7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8" gracePeriod=30
Jul 27 17:36:48.418695 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.418671    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:36:48.418894 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.418840    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" podUID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" containerName="payload-pre-processing" containerID="cri-o://ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0" gracePeriod=30
Jul 27 17:36:48.686519 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.686488    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:36:48.702214 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.702194    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:36:48.713119 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.713038    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:36:48.782051 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782012    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj6cv\" (UniqueName: \"kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv\") pod \"3c1f583d-681b-42b8-a346-119157b07d0f\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") "
Jul 27 17:36:48.782182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782073    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca\") pod \"3c1f583d-681b-42b8-a346-119157b07d0f\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") "
Jul 27 17:36:48.782182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782143    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j6ln\" (UniqueName: \"kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln\") pod \"2842ec04-fe56-46ce-9b19-bdf710638d7f\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") "
Jul 27 17:36:48.782182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782166    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw6zx\" (UniqueName: \"kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx\") pod \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") "
Jul 27 17:36:48.782360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782213    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume\") pod \"2842ec04-fe56-46ce-9b19-bdf710638d7f\" (UID: \"2842ec04-fe56-46ce-9b19-bdf710638d7f\") "
Jul 27 17:36:48.782360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782258    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume\") pod \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\" (UID: \"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f\") "
Jul 27 17:36:48.782360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782293    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls\") pod \"3c1f583d-681b-42b8-a346-119157b07d0f\" (UID: \"3c1f583d-681b-42b8-a346-119157b07d0f\") "
Jul 27 17:36:48.782562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782536    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c1f583d-681b-42b8-a346-119157b07d0f" (UID: "3c1f583d-681b-42b8-a346-119157b07d0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:36:48.782670 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2842ec04-fe56-46ce-9b19-bdf710638d7f" (UID: "2842ec04-fe56-46ce-9b19-bdf710638d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:36:48.782670 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.782638    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" (UID: "9bd9b054-40f4-4c9a-8ba5-e98d0b24744f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:36:48.784219 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.784194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln" (OuterVolumeSpecName: "kube-api-access-2j6ln") pod "2842ec04-fe56-46ce-9b19-bdf710638d7f" (UID: "2842ec04-fe56-46ce-9b19-bdf710638d7f"). InnerVolumeSpecName "kube-api-access-2j6ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:48.784325 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.784253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx" (OuterVolumeSpecName: "kube-api-access-zw6zx") pod "9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" (UID: "9bd9b054-40f4-4c9a-8ba5-e98d0b24744f"). InnerVolumeSpecName "kube-api-access-zw6zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:48.784325 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.784316    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv" (OuterVolumeSpecName: "kube-api-access-cj6cv") pod "3c1f583d-681b-42b8-a346-119157b07d0f" (UID: "3c1f583d-681b-42b8-a346-119157b07d0f"). InnerVolumeSpecName "kube-api-access-cj6cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:48.784484 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.784470    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c1f583d-681b-42b8-a346-119157b07d0f" (UID: "3c1f583d-681b-42b8-a346-119157b07d0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883348    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883370    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c1f583d-681b-42b8-a346-119157b07d0f-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883379    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6cv\" (UniqueName: \"kubernetes.io/projected/3c1f583d-681b-42b8-a346-119157b07d0f-kube-api-access-cj6cv\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883388    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1f583d-681b-42b8-a346-119157b07d0f-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883397    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6ln\" (UniqueName: \"kubernetes.io/projected/2842ec04-fe56-46ce-9b19-bdf710638d7f-kube-api-access-2j6ln\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883405    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6zx\" (UniqueName: \"kubernetes.io/projected/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f-kube-api-access-zw6zx\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:48.883674 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:48.883413    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2842ec04-fe56-46ce-9b19-bdf710638d7f-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:49.167208 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.167177    2564 generic.go:356] "Generic (PLEG): container finished" podID="2842ec04-fe56-46ce-9b19-bdf710638d7f" containerID="7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8" exitCode=0
Jul 27 17:36:49.167345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.167213    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"
Jul 27 17:36:49.167345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.167261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" event={"ID":"2842ec04-fe56-46ce-9b19-bdf710638d7f","Type":"ContainerDied","Data":"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"}
Jul 27 17:36:49.167345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.167313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75" event={"ID":"2842ec04-fe56-46ce-9b19-bdf710638d7f","Type":"ContainerDied","Data":"c0727d58201650ee413a4b86647113853d96859d4fd9cb9c1912e45ba64f3ef3"}
Jul 27 17:36:49.167345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.167340    2564 scope.go:117] "RemoveContainer" containerID="7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"
Jul 27 17:36:49.168573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.168552    2564 generic.go:356] "Generic (PLEG): container finished" podID="3c1f583d-681b-42b8-a346-119157b07d0f" containerID="4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830" exitCode=0
Jul 27 17:36:49.168675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.168584    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" event={"ID":"3c1f583d-681b-42b8-a346-119157b07d0f","Type":"ContainerDied","Data":"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"}
Jul 27 17:36:49.168675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.168599    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"
Jul 27 17:36:49.168675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.168636    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw" event={"ID":"3c1f583d-681b-42b8-a346-119157b07d0f","Type":"ContainerDied","Data":"ddd02f89df843e14ed8c3ea1554a91461e2b84148481714c535a30ffadf637a1"}
Jul 27 17:36:49.169916 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.169893    2564 generic.go:356] "Generic (PLEG): container finished" podID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" containerID="ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0" exitCode=0
Jul 27 17:36:49.170008 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.169933    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"
Jul 27 17:36:49.170008 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.169975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" event={"ID":"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f","Type":"ContainerDied","Data":"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"}
Jul 27 17:36:49.170120 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.170023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc" event={"ID":"9bd9b054-40f4-4c9a-8ba5-e98d0b24744f","Type":"ContainerDied","Data":"05861d266de903630a0a8526253001ea7a3c19a1dd9e269769254c3aaed10cdd"}
Jul 27 17:36:49.176076 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.176050    2564 scope.go:117] "RemoveContainer" containerID="7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"
Jul 27 17:36:49.176401 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:36:49.176376    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8\": container with ID starting with 7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8 not found: ID does not exist" containerID="7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"
Jul 27 17:36:49.176475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.176412    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8"} err="failed to get container status \"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8\": rpc error: code = NotFound desc = could not find container \"7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8\": container with ID starting with 7e330b4312f429990bbf2d6246da1ea7f4feece698d27374daa04db45f4c58b8 not found: ID does not exist"
Jul 27 17:36:49.176475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.176435    2564 scope.go:117] "RemoveContainer" containerID="4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"
Jul 27 17:36:49.187724 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.187701    2564 scope.go:117] "RemoveContainer" containerID="4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"
Jul 27 17:36:49.188204 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:36:49.188185    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830\": container with ID starting with 4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830 not found: ID does not exist" containerID="4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"
Jul 27 17:36:49.188295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.188213    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830"} err="failed to get container status \"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830\": rpc error: code = NotFound desc = could not find container \"4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830\": container with ID starting with 4350b79db4f3b47fb7aba516b7b05afd5a75f4bdc6b871e127a2b4b43c1a9830 not found: ID does not exist"
Jul 27 17:36:49.188295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.188234    2564 scope.go:117] "RemoveContainer" containerID="ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"
Jul 27 17:36:49.195258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.195244    2564 scope.go:117] "RemoveContainer" containerID="ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"
Jul 27 17:36:49.195506 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:36:49.195488    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0\": container with ID starting with ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0 not found: ID does not exist" containerID="ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"
Jul 27 17:36:49.195506 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.195511    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0"} err="failed to get container status \"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0\": rpc error: code = NotFound desc = could not find container \"ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0\": container with ID starting with ada2b07fb9f3ac30758594eb1dee1a2e04c1459f9348a82f3571fd7b87fda7a0 not found: ID does not exist"
Jul 27 17:36:49.197786 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.197766    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:36:49.203501 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.203483    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a6f598-66d99476c6-vlp75"]
Jul 27 17:36:49.215186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.215165    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:36:49.221121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.221103    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a6f598-7b99bf99d8-mmdtc"]
Jul 27 17:36:49.232652 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.232633    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:36:49.236482 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:49.236465    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a6f598-5c86f89849-b4ccw"]
Jul 27 17:36:50.659039 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:50.659009    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2842ec04-fe56-46ce-9b19-bdf710638d7f" path="/var/lib/kubelet/pods/2842ec04-fe56-46ce-9b19-bdf710638d7f/volumes"
Jul 27 17:36:50.659404 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:50.659371    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1f583d-681b-42b8-a346-119157b07d0f" path="/var/lib/kubelet/pods/3c1f583d-681b-42b8-a346-119157b07d0f/volumes"
Jul 27 17:36:50.659676 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:50.659663    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" path="/var/lib/kubelet/pods/9bd9b054-40f4-4c9a-8ba5-e98d0b24744f/volumes"
Jul 27 17:36:51.573216 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.573169    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" podUID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" containerName="main" containerID="cri-o://b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9" gracePeriod=45
Jul 27 17:36:51.819244 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.819222    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:36:51.907187 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907166    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907327 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907237    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907327 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907262    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907327 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907327 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907323    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk9cm\" (UniqueName: \"kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm\") pod \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\" (UID: \"77c17407-20f5-4a71-b1cf-6ea8e5de8201\") "
Jul 27 17:36:51.907591 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907545    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:51.907591 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907528    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home" (OuterVolumeSpecName: "home") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:51.907685 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907600    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache" (OuterVolumeSpecName: "model-cache") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:51.907728 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907686    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:51.907728 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907704    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:51.907728 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.907716    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:51.909411 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.909386    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:36:51.909411 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.909393    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm" (OuterVolumeSpecName: "dshm") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:51.909546 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:51.909448    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm" (OuterVolumeSpecName: "kube-api-access-gk9cm") pod "77c17407-20f5-4a71-b1cf-6ea8e5de8201" (UID: "77c17407-20f5-4a71-b1cf-6ea8e5de8201"). InnerVolumeSpecName "kube-api-access-gk9cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:52.008014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.007992    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9cm\" (UniqueName: \"kubernetes.io/projected/77c17407-20f5-4a71-b1cf-6ea8e5de8201-kube-api-access-gk9cm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.008014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.008012    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77c17407-20f5-4a71-b1cf-6ea8e5de8201-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.008135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.008022    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77c17407-20f5-4a71-b1cf-6ea8e5de8201-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.102556 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.102536    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:36:52.187192 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.187128    2564 generic.go:356] "Generic (PLEG): container finished" podID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" containerID="b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9" exitCode=0
Jul 27 17:36:52.187192 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.187166    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"
Jul 27 17:36:52.187329 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.187205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" event={"ID":"77c17407-20f5-4a71-b1cf-6ea8e5de8201","Type":"ContainerDied","Data":"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"}
Jul 27 17:36:52.187329 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.187242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl" event={"ID":"77c17407-20f5-4a71-b1cf-6ea8e5de8201","Type":"ContainerDied","Data":"0cf335b8833065254881f48aaa79508636fadd051faae4a4dbf94233d74c44cc"}
Jul 27 17:36:52.187329 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.187260    2564 scope.go:117] "RemoveContainer" containerID="b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"
Jul 27 17:36:52.202271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.202254    2564 scope.go:117] "RemoveContainer" containerID="b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"
Jul 27 17:36:52.202510 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:36:52.202493    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9\": container with ID starting with b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9 not found: ID does not exist" containerID="b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"
Jul 27 17:36:52.202575 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.202517    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9"} err="failed to get container status \"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9\": rpc error: code = NotFound desc = could not find container \"b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9\": container with ID starting with b5f7b4ccb6b40cd5e57ef017b03a31c9e81718073d571632766f886782449cd9 not found: ID does not exist"
Jul 27 17:36:52.214643 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.214621    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:36:52.218913 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.218871    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a6f598/test-model-a6f598-kserve-5b49b56b6c-mhztl"]
Jul 27 17:36:52.517963 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.517870    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" podUID="590296c0-81af-4102-9dba-2b8aa692fcc7" containerName="main" containerID="cri-o://c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef" gracePeriod=45
Jul 27 17:36:52.658103 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.658075    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" path="/var/lib/kubelet/pods/77c17407-20f5-4a71-b1cf-6ea8e5de8201/volumes"
Jul 27 17:36:52.767783 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.767761    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:52.815299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s964j\" (UniqueName: \"kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815304    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815360    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815400    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815450    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache\") pod \"590296c0-81af-4102-9dba-2b8aa692fcc7\" (UID: \"590296c0-81af-4102-9dba-2b8aa692fcc7\") "
Jul 27 17:36:52.815550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815511    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home" (OuterVolumeSpecName: "home") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:52.815677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:52.815735 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:52.815784 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815772    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.815839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815784    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.815839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.815792    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.817403 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.817384    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm" (OuterVolumeSpecName: "dshm") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:52.817492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.817472    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j" (OuterVolumeSpecName: "kube-api-access-s964j") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "kube-api-access-s964j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:52.817535 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.817511    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "590296c0-81af-4102-9dba-2b8aa692fcc7" (UID: "590296c0-81af-4102-9dba-2b8aa692fcc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:36:52.917069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.917046    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/590296c0-81af-4102-9dba-2b8aa692fcc7-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.917069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.917067    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590296c0-81af-4102-9dba-2b8aa692fcc7-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:52.917387 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:52.917078    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s964j\" (UniqueName: \"kubernetes.io/projected/590296c0-81af-4102-9dba-2b8aa692fcc7-kube-api-access-s964j\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:36:53.192783 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.192754    2564 generic.go:356] "Generic (PLEG): container finished" podID="590296c0-81af-4102-9dba-2b8aa692fcc7" containerID="c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef" exitCode=0
Jul 27 17:36:53.192938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.192796    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"
Jul 27 17:36:53.192938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.192836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" event={"ID":"590296c0-81af-4102-9dba-2b8aa692fcc7","Type":"ContainerDied","Data":"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"}
Jul 27 17:36:53.192938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.192874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2" event={"ID":"590296c0-81af-4102-9dba-2b8aa692fcc7","Type":"ContainerDied","Data":"88c2bc7aacfee923e486336f14ba2d3bc76c1ea5c4b906f1ff1a398cee73a997"}
Jul 27 17:36:53.192938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.192902    2564 scope.go:117] "RemoveContainer" containerID="c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"
Jul 27 17:36:53.201921 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.201906    2564 scope.go:117] "RemoveContainer" containerID="c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"
Jul 27 17:36:53.202145 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:36:53.202128    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef\": container with ID starting with c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef not found: ID does not exist" containerID="c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"
Jul 27 17:36:53.202192 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.202154    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef"} err="failed to get container status \"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef\": rpc error: code = NotFound desc = could not find container \"c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef\": container with ID starting with c4f7e8440ef07798e4abbbdb1867a4345fa82a4d17d96fa378abe0e835a10eef not found: ID does not exist"
Jul 27 17:36:53.215485 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.215458    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:36:53.219675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:53.219656    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7a55cb/test-model-7a55cb-kserve-57c5999fff-bbnh2"]
Jul 27 17:36:54.658273 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:36:54.658240    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590296c0-81af-4102-9dba-2b8aa692fcc7" path="/var/lib/kubelet/pods/590296c0-81af-4102-9dba-2b8aa692fcc7/volumes"
Jul 27 17:37:10.295334 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.295299    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:37:10.295945 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.295544    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" podUID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" containerName="maas-api" containerID="cri-o://5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c" gracePeriod=30
Jul 27 17:37:10.365412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.365377    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:37:10.365628 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.365605    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" podUID="a8e792b4-b025-44b0-8efa-32a23ede578d" containerName="payload-processing" containerID="cri-o://d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891" gracePeriod=30
Jul 27 17:37:10.622068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.622046    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:37:10.624936 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.624918    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:37:10.765562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.765533    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca\") pod \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") "
Jul 27 17:37:10.765727 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.765587    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5zlr\" (UniqueName: \"kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr\") pod \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") "
Jul 27 17:37:10.765727 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.765611    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") pod \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\" (UID: \"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae\") "
Jul 27 17:37:10.765727 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.765635    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpx6g\" (UniqueName: \"kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g\") pod \"a8e792b4-b025-44b0-8efa-32a23ede578d\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") "
Jul 27 17:37:10.765727 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.765678    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume\") pod \"a8e792b4-b025-44b0-8efa-32a23ede578d\" (UID: \"a8e792b4-b025-44b0-8efa-32a23ede578d\") "
Jul 27 17:37:10.766084 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.766050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" (UID: "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:37:10.766203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.766176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8e792b4-b025-44b0-8efa-32a23ede578d" (UID: "a8e792b4-b025-44b0-8efa-32a23ede578d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:37:10.767856 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.767827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g" (OuterVolumeSpecName: "kube-api-access-zpx6g") pod "a8e792b4-b025-44b0-8efa-32a23ede578d" (UID: "a8e792b4-b025-44b0-8efa-32a23ede578d"). InnerVolumeSpecName "kube-api-access-zpx6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:10.767856 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.767827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr" (OuterVolumeSpecName: "kube-api-access-s5zlr") pod "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" (UID: "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae"). InnerVolumeSpecName "kube-api-access-s5zlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:10.768023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.767864    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" (UID: "425cabe6-cb2e-4e34-8c2f-ad44f28de6ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:37:10.867163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.867092    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e792b4-b025-44b0-8efa-32a23ede578d-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:37:10.867163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.867117    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:37:10.867163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.867128    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5zlr\" (UniqueName: \"kubernetes.io/projected/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-kube-api-access-s5zlr\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:37:10.867163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.867137    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:37:10.867163 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:10.867145    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpx6g\" (UniqueName: \"kubernetes.io/projected/a8e792b4-b025-44b0-8efa-32a23ede578d-kube-api-access-zpx6g\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:37:11.276546 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.276511    2564 generic.go:356] "Generic (PLEG): container finished" podID="a8e792b4-b025-44b0-8efa-32a23ede578d" containerID="d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891" exitCode=0
Jul 27 17:37:11.276730 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.276564    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"
Jul 27 17:37:11.276730 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.276594    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" event={"ID":"a8e792b4-b025-44b0-8efa-32a23ede578d","Type":"ContainerDied","Data":"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"}
Jul 27 17:37:11.276730 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.276645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546" event={"ID":"a8e792b4-b025-44b0-8efa-32a23ede578d","Type":"ContainerDied","Data":"7f0461ac51478b45e478176b9af9f8271d764ba798958675aab03451faf06ed7"}
Jul 27 17:37:11.276730 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.276666    2564 scope.go:117] "RemoveContainer" containerID="d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"
Jul 27 17:37:11.277968 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.277941    2564 generic.go:356] "Generic (PLEG): container finished" podID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" containerID="5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c" exitCode=0
Jul 27 17:37:11.278080 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.277989    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"
Jul 27 17:37:11.278080 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.278018    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" event={"ID":"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae","Type":"ContainerDied","Data":"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"}
Jul 27 17:37:11.278080 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.278074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm" event={"ID":"425cabe6-cb2e-4e34-8c2f-ad44f28de6ae","Type":"ContainerDied","Data":"2bd3c747007397447c55532545036fd9b4b470050f95b67f1bddc97e20aacd58"}
Jul 27 17:37:11.285447 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.285431    2564 scope.go:117] "RemoveContainer" containerID="d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"
Jul 27 17:37:11.285725 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:37:11.285698    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891\": container with ID starting with d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891 not found: ID does not exist" containerID="d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"
Jul 27 17:37:11.285834 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.285732    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891"} err="failed to get container status \"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891\": rpc error: code = NotFound desc = could not find container \"d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891\": container with ID starting with d43aba7ec0f1e8d263a5ed4145880b5871d159618d385af814d82b39b1691891 not found: ID does not exist"
Jul 27 17:37:11.285834 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.285755    2564 scope.go:117] "RemoveContainer" containerID="5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"
Jul 27 17:37:11.293550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.293534    2564 scope.go:117] "RemoveContainer" containerID="5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"
Jul 27 17:37:11.293794 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:37:11.293775    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c\": container with ID starting with 5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c not found: ID does not exist" containerID="5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"
Jul 27 17:37:11.293853 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.293800    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c"} err="failed to get container status \"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c\": rpc error: code = NotFound desc = could not find container \"5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c\": container with ID starting with 5fe6c6002bfc57338ba359d3020d148a1a001e2ed3882f7aa502db638e80f25c not found: ID does not exist"
Jul 27 17:37:11.302993 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.302974    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:37:11.307203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.307181    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7a55cb-bf674b6b-967fm"]
Jul 27 17:37:11.319765 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.319744    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:37:11.323657 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:11.323635    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7a55cb-78668d574b-g6546"]
Jul 27 17:37:12.657594 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:12.657563    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" path="/var/lib/kubelet/pods/425cabe6-cb2e-4e34-8c2f-ad44f28de6ae/volumes"
Jul 27 17:37:12.657965 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:12.657910    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e792b4-b025-44b0-8efa-32a23ede578d" path="/var/lib/kubelet/pods/a8e792b4-b025-44b0-8efa-32a23ede578d/volumes"
Jul 27 17:37:12.997622 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:12.997542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:37:25.845293 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845237    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:37:25.845739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845723    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1f583d-681b-42b8-a346-119157b07d0f" containerName="maas-api"
Jul 27 17:37:25.845788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845743    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1f583d-681b-42b8-a346-119157b07d0f" containerName="maas-api"
Jul 27 17:37:25.845788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845756    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" containerName="maas-api"
Jul 27 17:37:25.845788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845765    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" containerName="maas-api"
Jul 27 17:37:25.845788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845784    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590296c0-81af-4102-9dba-2b8aa692fcc7" containerName="main"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845793    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="590296c0-81af-4102-9dba-2b8aa692fcc7" containerName="main"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" containerName="main"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845814    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" containerName="main"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845826    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" containerName="payload-pre-processing"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845835    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" containerName="payload-pre-processing"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845851    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2842ec04-fe56-46ce-9b19-bdf710638d7f" containerName="payload-processing"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845860    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2842ec04-fe56-46ce-9b19-bdf710638d7f" containerName="payload-processing"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845873    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e792b4-b025-44b0-8efa-32a23ede578d" containerName="payload-processing"
Jul 27 17:37:25.845934 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845896    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e792b4-b025-44b0-8efa-32a23ede578d" containerName="payload-processing"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845977    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="425cabe6-cb2e-4e34-8c2f-ad44f28de6ae" containerName="maas-api"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.845990    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="590296c0-81af-4102-9dba-2b8aa692fcc7" containerName="main"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.846001    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1f583d-681b-42b8-a346-119157b07d0f" containerName="maas-api"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.846010    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2842ec04-fe56-46ce-9b19-bdf710638d7f" containerName="payload-processing"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.846020    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd9b054-40f4-4c9a-8ba5-e98d0b24744f" containerName="payload-pre-processing"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.846028    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e792b4-b025-44b0-8efa-32a23ede578d" containerName="payload-processing"
Jul 27 17:37:25.846222 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.846036    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c17407-20f5-4a71-b1cf-6ea8e5de8201" containerName="main"
Jul 27 17:37:25.850983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.850965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.853854 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.853831    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:37:25.853854 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.853845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:37:25.854037 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.853850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:37:25.854037 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.853835    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f6e958-serving-cert\""
Jul 27 17:37:25.857340 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.857307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:37:25.891083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.891052    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.891186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.891093    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6zk\" (UniqueName: \"kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.891186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.891111    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.992358 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.992332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.992509 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.992368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6zk\" (UniqueName: \"kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.992509 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.992393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.993071 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.993049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:25.994761 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:25.994742    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:26.001146 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.001127    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6zk\" (UniqueName: \"kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk\") pod \"maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:26.162119 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.162091    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:26.293491 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.293439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:37:26.295802 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:26.295774    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d5eec6_d58f_4b5c_9dc8_8b223a3221f2.slice/crio-7d8ef17fe161106f4592f9327a403c14a41710a0577abbb58f8251300e975143 WatchSource:0}: Error finding container 7d8ef17fe161106f4592f9327a403c14a41710a0577abbb58f8251300e975143: Status 404 returned error can't find the container with id 7d8ef17fe161106f4592f9327a403c14a41710a0577abbb58f8251300e975143
Jul 27 17:37:26.297187 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.297167    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:37:26.355123 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.355098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" event={"ID":"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2","Type":"ContainerStarted","Data":"7d8ef17fe161106f4592f9327a403c14a41710a0577abbb58f8251300e975143"}
Jul 27 17:37:26.588170 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.588094    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:37:26.592554 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.592537    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:37:26.592671 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.592653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.595754 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.595735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f6e958\""
Jul 27 17:37:26.596534 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.596515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.603384 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.603332    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:37:26.604731 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.604713    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:37:26.698563 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.698531    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbnz\" (UniqueName: \"kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.698739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.698664    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8qm\" (UniqueName: \"kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.698792 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.698750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.698792 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.698779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.799791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.799766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8qm\" (UniqueName: \"kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.799911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.799818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.799911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.799838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.799911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.799866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbnz\" (UniqueName: \"kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.800684 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.800663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.800724 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.800663    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.808674 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.808650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8qm\" (UniqueName: \"kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm\") pod \"payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.808762 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.808727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbnz\" (UniqueName: \"kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz\") pod \"payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:26.905849 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.905819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:26.912895 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:26.912860    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:27.056734 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.056707    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:37:27.058649 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:27.058603    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67966780_c88f_4db9_9f80_ac5731be44be.slice/crio-43c544844104fce05969dabfaba7a90e327fc1ba085433ebeff37e25afb81c13 WatchSource:0}: Error finding container 43c544844104fce05969dabfaba7a90e327fc1ba085433ebeff37e25afb81c13: Status 404 returned error can't find the container with id 43c544844104fce05969dabfaba7a90e327fc1ba085433ebeff37e25afb81c13
Jul 27 17:37:27.079278 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.079257    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:37:27.081230 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:27.081200    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8740860d_308a_4b38_934e_20f782992095.slice/crio-8d37d5e3d3c2c44ca8903b8f68a9b0fa5326cfd2a9ca3079a8c92efe84c2b74c WatchSource:0}: Error finding container 8d37d5e3d3c2c44ca8903b8f68a9b0fa5326cfd2a9ca3079a8c92efe84c2b74c: Status 404 returned error can't find the container with id 8d37d5e3d3c2c44ca8903b8f68a9b0fa5326cfd2a9ca3079a8c92efe84c2b74c
Jul 27 17:37:27.360765 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.360673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" event={"ID":"8740860d-308a-4b38-934e-20f782992095","Type":"ContainerStarted","Data":"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"}
Jul 27 17:37:27.360765 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.360725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" event={"ID":"8740860d-308a-4b38-934e-20f782992095","Type":"ContainerStarted","Data":"8d37d5e3d3c2c44ca8903b8f68a9b0fa5326cfd2a9ca3079a8c92efe84c2b74c"}
Jul 27 17:37:27.360991 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.360775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:27.362185 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.362164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" event={"ID":"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2","Type":"ContainerStarted","Data":"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"}
Jul 27 17:37:27.362262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.362207    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:27.363489 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.363463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" event={"ID":"67966780-c88f-4db9-9f80-ac5731be44be","Type":"ContainerStarted","Data":"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"}
Jul 27 17:37:27.363627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.363505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" event={"ID":"67966780-c88f-4db9-9f80-ac5731be44be","Type":"ContainerStarted","Data":"43c544844104fce05969dabfaba7a90e327fc1ba085433ebeff37e25afb81c13"}
Jul 27 17:37:27.363627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.363604    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:27.381020 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.380975    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" podStartSLOduration=1.380962608 podStartE2EDuration="1.380962608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:27.379102844 +0000 UTC m=+2467.211467831" watchObservedRunningTime="2026-07-27 17:37:27.380962608 +0000 UTC m=+2467.213327562"
Jul 27 17:37:27.397909 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.397855    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" podStartSLOduration=1.976108441 podStartE2EDuration="2.397846174s" totalImagesPullingTime="421.737733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:27.396802555 +0000 UTC m=+2467.229167549" watchObservedRunningTime="2026-07-27 17:37:27.397846174 +0000 UTC m=+2467.230211195"
Jul 27 17:37:27.416413 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:27.416376    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" podStartSLOduration=1.4163662750000001 podStartE2EDuration="1.416366275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:27.41510455 +0000 UTC m=+2467.247469545" watchObservedRunningTime="2026-07-27 17:37:27.416366275 +0000 UTC m=+2467.248731242"
Jul 27 17:37:30.996543 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:30.996515    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:37:33.373566 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.373538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:37:33.838245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.838212    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:37:33.841687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.841669    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.845145 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.845124    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-05e104-serving-cert\""
Jul 27 17:37:33.853552 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.853518    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:37:33.854919 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.854898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.855023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.854975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.858897 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.855368    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9r4\" (UniqueName: \"kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.956864 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.956835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.957026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.956868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9r4\" (UniqueName: \"kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.957026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.956932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.957026 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:37:33.956982    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-serving-cert: secret "maas-api-e2e-api-b-05e104-serving-cert" not found
Jul 27 17:37:33.957192 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:37:33.957059    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls podName:6be63f3d-6e6b-44e6-9c14-4bb552572e48 nodeName:}" failed. No retries permitted until 2026-07-27 17:37:34.45703904 +0000 UTC m=+2474.289404013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls") pod "maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" (UID: "6be63f3d-6e6b-44e6-9c14-4bb552572e48") : secret "maas-api-e2e-api-b-05e104-serving-cert" not found
Jul 27 17:37:33.957562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.957543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:33.968184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:33.968159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9r4\" (UniqueName: \"kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:34.461210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.461177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:34.463473 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.463457    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") pod \"maas-api-e2e-api-b-05e104-79747d44bf-jr7hl\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:34.620857 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.620824    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:37:34.624403 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.624380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.627729 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.627704    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-05e104\""
Jul 27 17:37:34.633074 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.633040    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:37:34.638022 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.638002    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:37:34.638136 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.638118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.640965 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.640946    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-05e104-dockercfg-9jsvt\""
Jul 27 17:37:34.647520 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.647500    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:37:34.662779 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.662745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.662912 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.662781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86z2m\" (UniqueName: \"kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.662912 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.662835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8j4s\" (UniqueName: \"kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.663027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.662937    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.751747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.751681    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:34.763901 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.763859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.764002 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.763941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.764002 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.763968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86z2m\" (UniqueName: \"kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.764090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.764064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8j4s\" (UniqueName: \"kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.764724 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.764704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.764908 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.764871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.774719 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.774696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8j4s\" (UniqueName: \"kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s\") pod \"payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.774861 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.774842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86z2m\" (UniqueName: \"kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m\") pod \"payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") " pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:34.884173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.884151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:37:34.886018 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:34.885995    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be63f3d_6e6b_44e6_9c14_4bb552572e48.slice/crio-150fdeef7ec48eb59f96d9f6386712620c6262e244b970802bf634ec9e81dc21 WatchSource:0}: Error finding container 150fdeef7ec48eb59f96d9f6386712620c6262e244b970802bf634ec9e81dc21: Status 404 returned error can't find the container with id 150fdeef7ec48eb59f96d9f6386712620c6262e244b970802bf634ec9e81dc21
Jul 27 17:37:34.938373 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.938347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:34.950381 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:34.950363    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:35.096052 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.095977    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:37:35.098914 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:35.098867    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30400a06_d0dc_424d_aa74_e725e7948e1c.slice/crio-36605a0ece7188c2c8fd4055f72bdf4108d82e7c7be191d59a10f8b6890d41a6 WatchSource:0}: Error finding container 36605a0ece7188c2c8fd4055f72bdf4108d82e7c7be191d59a10f8b6890d41a6: Status 404 returned error can't find the container with id 36605a0ece7188c2c8fd4055f72bdf4108d82e7c7be191d59a10f8b6890d41a6
Jul 27 17:37:35.126201 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.126178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:37:35.137975 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:37:35.137947    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88291ae4_4a7b_485e_a823_bf29573135ea.slice/crio-9d444ac483addcee5ba80fd61a54e65e739d9a1ecab1820f34281a8fdf9c2f03 WatchSource:0}: Error finding container 9d444ac483addcee5ba80fd61a54e65e739d9a1ecab1820f34281a8fdf9c2f03: Status 404 returned error can't find the container with id 9d444ac483addcee5ba80fd61a54e65e739d9a1ecab1820f34281a8fdf9c2f03
Jul 27 17:37:35.398767 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.398732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" event={"ID":"6be63f3d-6e6b-44e6-9c14-4bb552572e48","Type":"ContainerStarted","Data":"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"}
Jul 27 17:37:35.398951 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.398796    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:35.398951 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.398816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" event={"ID":"6be63f3d-6e6b-44e6-9c14-4bb552572e48","Type":"ContainerStarted","Data":"150fdeef7ec48eb59f96d9f6386712620c6262e244b970802bf634ec9e81dc21"}
Jul 27 17:37:35.400252 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.400222    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" event={"ID":"30400a06-d0dc-424d-aa74-e725e7948e1c","Type":"ContainerStarted","Data":"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"}
Jul 27 17:37:35.400347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.400271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" event={"ID":"30400a06-d0dc-424d-aa74-e725e7948e1c","Type":"ContainerStarted","Data":"36605a0ece7188c2c8fd4055f72bdf4108d82e7c7be191d59a10f8b6890d41a6"}
Jul 27 17:37:35.400347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.400299    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:37:35.401601 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.401577    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" event={"ID":"88291ae4-4a7b-485e-a823-bf29573135ea","Type":"ContainerStarted","Data":"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"}
Jul 27 17:37:35.401687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.401622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" event={"ID":"88291ae4-4a7b-485e-a823-bf29573135ea","Type":"ContainerStarted","Data":"9d444ac483addcee5ba80fd61a54e65e739d9a1ecab1820f34281a8fdf9c2f03"}
Jul 27 17:37:35.401687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.401651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:35.418219 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.418180    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" podStartSLOduration=2.090031314 podStartE2EDuration="2.418166789s" totalImagesPullingTime="328.135475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:35.416198979 +0000 UTC m=+2475.248563965" watchObservedRunningTime="2026-07-27 17:37:35.418166789 +0000 UTC m=+2475.250531765"
Jul 27 17:37:35.436302 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.436261    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" podStartSLOduration=1.436247112 podStartE2EDuration="1.436247112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:35.434683845 +0000 UTC m=+2475.267048852" watchObservedRunningTime="2026-07-27 17:37:35.436247112 +0000 UTC m=+2475.268612088"
Jul 27 17:37:35.458744 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:35.458701    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" podStartSLOduration=1.458688191 podStartE2EDuration="1.458688191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:35.456901898 +0000 UTC m=+2475.289266891" watchObservedRunningTime="2026-07-27 17:37:35.458688191 +0000 UTC m=+2475.291053167"
Jul 27 17:37:38.370949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:38.370906    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:37:38.371391 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:38.371261    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:37:41.410544 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:41.410514    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:37:42.687979 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:42.687947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:37:46.407471 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:46.407438    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:37:46.407904 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:37:46.407513    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:38:01.148782 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.148751    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:38:01.149265 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.148991    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" podUID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" containerName="maas-api" containerID="cri-o://cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748" gracePeriod=30
Jul 27 17:38:01.210121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.210083    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:38:01.211700 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.210312    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" podUID="8740860d-308a-4b38-934e-20f782992095" containerName="payload-processing" containerID="cri-o://743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30" gracePeriod=30
Jul 27 17:38:01.214032 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.213479    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:38:01.214032 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.213730    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" podUID="67966780-c88f-4db9-9f80-ac5731be44be" containerName="payload-pre-processing" containerID="cri-o://25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50" gracePeriod=30
Jul 27 17:38:01.459926 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.459898    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:38:01.486238 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.486212    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:38:01.491598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.491566    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:38:01.533594 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.533560    2564 generic.go:356] "Generic (PLEG): container finished" podID="8740860d-308a-4b38-934e-20f782992095" containerID="743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30" exitCode=0
Jul 27 17:38:01.533763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.533600    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"
Jul 27 17:38:01.533763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.533656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" event={"ID":"8740860d-308a-4b38-934e-20f782992095","Type":"ContainerDied","Data":"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"}
Jul 27 17:38:01.533763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.533711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56" event={"ID":"8740860d-308a-4b38-934e-20f782992095","Type":"ContainerDied","Data":"8d37d5e3d3c2c44ca8903b8f68a9b0fa5326cfd2a9ca3079a8c92efe84c2b74c"}
Jul 27 17:38:01.533763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.533739    2564 scope.go:117] "RemoveContainer" containerID="743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"
Jul 27 17:38:01.535140 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.535108    2564 generic.go:356] "Generic (PLEG): container finished" podID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" containerID="cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748" exitCode=0
Jul 27 17:38:01.535262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.535154    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"
Jul 27 17:38:01.535262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.535196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" event={"ID":"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2","Type":"ContainerDied","Data":"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"}
Jul 27 17:38:01.535262 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.535253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g" event={"ID":"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2","Type":"ContainerDied","Data":"7d8ef17fe161106f4592f9327a403c14a41710a0577abbb58f8251300e975143"}
Jul 27 17:38:01.536566 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.536546    2564 generic.go:356] "Generic (PLEG): container finished" podID="67966780-c88f-4db9-9f80-ac5731be44be" containerID="25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50" exitCode=0
Jul 27 17:38:01.536677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.536572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" event={"ID":"67966780-c88f-4db9-9f80-ac5731be44be","Type":"ContainerDied","Data":"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"}
Jul 27 17:38:01.536677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.536597    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"
Jul 27 17:38:01.536677 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.536603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p" event={"ID":"67966780-c88f-4db9-9f80-ac5731be44be","Type":"ContainerDied","Data":"43c544844104fce05969dabfaba7a90e327fc1ba085433ebeff37e25afb81c13"}
Jul 27 17:38:01.544973 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.544949    2564 scope.go:117] "RemoveContainer" containerID="743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"
Jul 27 17:38:01.545266 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:01.545247    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30\": container with ID starting with 743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30 not found: ID does not exist" containerID="743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"
Jul 27 17:38:01.545337 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.545278    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30"} err="failed to get container status \"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30\": rpc error: code = NotFound desc = could not find container \"743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30\": container with ID starting with 743c8ebac0a58864c2ad92205830e7f028a5dd7efc16e679e7d9e1511f50cd30 not found: ID does not exist"
Jul 27 17:38:01.545337 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.545295    2564 scope.go:117] "RemoveContainer" containerID="cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"
Jul 27 17:38:01.552819 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.552802    2564 scope.go:117] "RemoveContainer" containerID="cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"
Jul 27 17:38:01.553065 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:01.553046    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748\": container with ID starting with cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748 not found: ID does not exist" containerID="cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"
Jul 27 17:38:01.553134 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.553070    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748"} err="failed to get container status \"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748\": rpc error: code = NotFound desc = could not find container \"cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748\": container with ID starting with cb9f275d16adef2987bc7338977e33b9d010d517610b878c0eb03acab1974748 not found: ID does not exist"
Jul 27 17:38:01.553134 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.553086    2564 scope.go:117] "RemoveContainer" containerID="25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"
Jul 27 17:38:01.560578 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.560560    2564 scope.go:117] "RemoveContainer" containerID="25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"
Jul 27 17:38:01.560821 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:01.560802    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50\": container with ID starting with 25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50 not found: ID does not exist" containerID="25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"
Jul 27 17:38:01.560865 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.560828    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50"} err="failed to get container status \"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50\": rpc error: code = NotFound desc = could not find container \"25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50\": container with ID starting with 25e21c2e5d0cebd59f74a43c9b847b3ceaefb9512f287c8de18240651027af50 not found: ID does not exist"
Jul 27 17:38:01.581120 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca\") pod \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") "
Jul 27 17:38:01.581190 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb8qm\" (UniqueName: \"kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm\") pod \"67966780-c88f-4db9-9f80-ac5731be44be\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") "
Jul 27 17:38:01.581190 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581175    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume\") pod \"8740860d-308a-4b38-934e-20f782992095\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") "
Jul 27 17:38:01.581279 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqbnz\" (UniqueName: \"kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz\") pod \"8740860d-308a-4b38-934e-20f782992095\" (UID: \"8740860d-308a-4b38-934e-20f782992095\") "
Jul 27 17:38:01.581279 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581223    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls\") pod \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") "
Jul 27 17:38:01.581279 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581244    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume\") pod \"67966780-c88f-4db9-9f80-ac5731be44be\" (UID: \"67966780-c88f-4db9-9f80-ac5731be44be\") "
Jul 27 17:38:01.581433 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581294    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt6zk\" (UniqueName: \"kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk\") pod \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\" (UID: \"16d5eec6-d58f-4b5c-9dc8-8b223a3221f2\") "
Jul 27 17:38:01.581489 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" (UID: "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:01.581630 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581602    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8740860d-308a-4b38-934e-20f782992095" (UID: "8740860d-308a-4b38-934e-20f782992095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:01.581702 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581670    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67966780-c88f-4db9-9f80-ac5731be44be" (UID: "67966780-c88f-4db9-9f80-ac5731be44be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:01.581702 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581685    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.581810 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.581704    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8740860d-308a-4b38-934e-20f782992095-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.583276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.583243    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" (UID: "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:38:01.583373 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.583317    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz" (OuterVolumeSpecName: "kube-api-access-gqbnz") pod "8740860d-308a-4b38-934e-20f782992095" (UID: "8740860d-308a-4b38-934e-20f782992095"). InnerVolumeSpecName "kube-api-access-gqbnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:01.583373 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.583343    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm" (OuterVolumeSpecName: "kube-api-access-qb8qm") pod "67966780-c88f-4db9-9f80-ac5731be44be" (UID: "67966780-c88f-4db9-9f80-ac5731be44be"). InnerVolumeSpecName "kube-api-access-qb8qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:01.583455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.583424    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk" (OuterVolumeSpecName: "kube-api-access-bt6zk") pod "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" (UID: "16d5eec6-d58f-4b5c-9dc8-8b223a3221f2"). InnerVolumeSpecName "kube-api-access-bt6zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:01.682747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.682695    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6zk\" (UniqueName: \"kubernetes.io/projected/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-kube-api-access-bt6zk\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.682747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.682713    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8qm\" (UniqueName: \"kubernetes.io/projected/67966780-c88f-4db9-9f80-ac5731be44be-kube-api-access-qb8qm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.682747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.682723    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqbnz\" (UniqueName: \"kubernetes.io/projected/8740860d-308a-4b38-934e-20f782992095-kube-api-access-gqbnz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.682747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.682733    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.682747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.682742    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67966780-c88f-4db9-9f80-ac5731be44be-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:01.863745 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.863718    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:38:01.867559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.867538    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6e958-74ddcdb778-nfh56"]
Jul 27 17:38:01.879869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.879848    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:38:01.884827 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.884807    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6e958-54d5fc8948-mgw8g"]
Jul 27 17:38:01.896547 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.896525    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:38:01.900496 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:01.900476    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6e958-576f598bb8-fd77p"]
Jul 27 17:38:02.658214 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:02.658186    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" path="/var/lib/kubelet/pods/16d5eec6-d58f-4b5c-9dc8-8b223a3221f2/volumes"
Jul 27 17:38:02.658562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:02.658504    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67966780-c88f-4db9-9f80-ac5731be44be" path="/var/lib/kubelet/pods/67966780-c88f-4db9-9f80-ac5731be44be/volumes"
Jul 27 17:38:02.658804 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:02.658791    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8740860d-308a-4b38-934e-20f782992095" path="/var/lib/kubelet/pods/8740860d-308a-4b38-934e-20f782992095/volumes"
Jul 27 17:38:03.607978 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:03.607944    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:38:28.799833 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.799800    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:38:28.800354 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.800058    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" podUID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" containerName="maas-api" containerID="cri-o://7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07" gracePeriod=30
Jul 27 17:38:28.853848 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.853818    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:38:28.854082 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.854057    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" podUID="88291ae4-4a7b-485e-a823-bf29573135ea" containerName="payload-processing" containerID="cri-o://144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e" gracePeriod=30
Jul 27 17:38:28.861126 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.861101    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:38:28.861368 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:28.861290    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" podUID="30400a06-d0dc-424d-aa74-e725e7948e1c" containerName="payload-pre-processing" containerID="cri-o://f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4" gracePeriod=30
Jul 27 17:38:29.118641 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.118618    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:38:29.128349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.128325    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:38:29.139062 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.139042    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:38:29.209466 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209442    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8j4s\" (UniqueName: \"kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s\") pod \"30400a06-d0dc-424d-aa74-e725e7948e1c\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") "
Jul 27 17:38:29.209618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209477    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume\") pod \"88291ae4-4a7b-485e-a823-bf29573135ea\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") "
Jul 27 17:38:29.209618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209508    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca\") pod \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") "
Jul 27 17:38:29.209618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209522    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx9r4\" (UniqueName: \"kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4\") pod \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") "
Jul 27 17:38:29.209802 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume\") pod \"30400a06-d0dc-424d-aa74-e725e7948e1c\" (UID: \"30400a06-d0dc-424d-aa74-e725e7948e1c\") "
Jul 27 17:38:29.209802 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209776    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") pod \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\" (UID: \"6be63f3d-6e6b-44e6-9c14-4bb552572e48\") "
Jul 27 17:38:29.209957 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86z2m\" (UniqueName: \"kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m\") pod \"88291ae4-4a7b-485e-a823-bf29573135ea\" (UID: \"88291ae4-4a7b-485e-a823-bf29573135ea\") "
Jul 27 17:38:29.209957 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.209843    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6be63f3d-6e6b-44e6-9c14-4bb552572e48" (UID: "6be63f3d-6e6b-44e6-9c14-4bb552572e48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:29.210075 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.210009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88291ae4-4a7b-485e-a823-bf29573135ea" (UID: "88291ae4-4a7b-485e-a823-bf29573135ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:29.210139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.210078    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30400a06-d0dc-424d-aa74-e725e7948e1c" (UID: "30400a06-d0dc-424d-aa74-e725e7948e1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:38:29.210139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.210089    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88291ae4-4a7b-485e-a823-bf29573135ea-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.210139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.210123    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be63f3d-6e6b-44e6-9c14-4bb552572e48-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.211549 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.211523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s" (OuterVolumeSpecName: "kube-api-access-x8j4s") pod "30400a06-d0dc-424d-aa74-e725e7948e1c" (UID: "30400a06-d0dc-424d-aa74-e725e7948e1c"). InnerVolumeSpecName "kube-api-access-x8j4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:29.211757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.211738    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m" (OuterVolumeSpecName: "kube-api-access-86z2m") pod "88291ae4-4a7b-485e-a823-bf29573135ea" (UID: "88291ae4-4a7b-485e-a823-bf29573135ea"). InnerVolumeSpecName "kube-api-access-86z2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:29.212189 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.212160    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6be63f3d-6e6b-44e6-9c14-4bb552572e48" (UID: "6be63f3d-6e6b-44e6-9c14-4bb552572e48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:38:29.212189 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.212171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4" (OuterVolumeSpecName: "kube-api-access-fx9r4") pod "6be63f3d-6e6b-44e6-9c14-4bb552572e48" (UID: "6be63f3d-6e6b-44e6-9c14-4bb552572e48"). InnerVolumeSpecName "kube-api-access-fx9r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:38:29.310799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.310728    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8j4s\" (UniqueName: \"kubernetes.io/projected/30400a06-d0dc-424d-aa74-e725e7948e1c-kube-api-access-x8j4s\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.310799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.310761    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx9r4\" (UniqueName: \"kubernetes.io/projected/6be63f3d-6e6b-44e6-9c14-4bb552572e48-kube-api-access-fx9r4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.310799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.310771    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30400a06-d0dc-424d-aa74-e725e7948e1c-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.310799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.310780    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be63f3d-6e6b-44e6-9c14-4bb552572e48-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.310799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.310789    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86z2m\" (UniqueName: \"kubernetes.io/projected/88291ae4-4a7b-485e-a823-bf29573135ea-kube-api-access-86z2m\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:38:29.667386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.667354    2564 generic.go:356] "Generic (PLEG): container finished" podID="30400a06-d0dc-424d-aa74-e725e7948e1c" containerID="f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4" exitCode=0
Jul 27 17:38:29.667559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.667398    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"
Jul 27 17:38:29.667559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.667434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" event={"ID":"30400a06-d0dc-424d-aa74-e725e7948e1c","Type":"ContainerDied","Data":"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"}
Jul 27 17:38:29.667559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.667484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk" event={"ID":"30400a06-d0dc-424d-aa74-e725e7948e1c","Type":"ContainerDied","Data":"36605a0ece7188c2c8fd4055f72bdf4108d82e7c7be191d59a10f8b6890d41a6"}
Jul 27 17:38:29.667559 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.667509    2564 scope.go:117] "RemoveContainer" containerID="f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"
Jul 27 17:38:29.668788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.668763    2564 generic.go:356] "Generic (PLEG): container finished" podID="88291ae4-4a7b-485e-a823-bf29573135ea" containerID="144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e" exitCode=0
Jul 27 17:38:29.668896 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.668854    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"
Jul 27 17:38:29.668896 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.668870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" event={"ID":"88291ae4-4a7b-485e-a823-bf29573135ea","Type":"ContainerDied","Data":"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"}
Jul 27 17:38:29.669087 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.668931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv" event={"ID":"88291ae4-4a7b-485e-a823-bf29573135ea","Type":"ContainerDied","Data":"9d444ac483addcee5ba80fd61a54e65e739d9a1ecab1820f34281a8fdf9c2f03"}
Jul 27 17:38:29.670246 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.670223    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" containerID="7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07" exitCode=0
Jul 27 17:38:29.670352 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.670253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" event={"ID":"6be63f3d-6e6b-44e6-9c14-4bb552572e48","Type":"ContainerDied","Data":"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"}
Jul 27 17:38:29.670352 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.670274    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"
Jul 27 17:38:29.670352 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.670296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl" event={"ID":"6be63f3d-6e6b-44e6-9c14-4bb552572e48","Type":"ContainerDied","Data":"150fdeef7ec48eb59f96d9f6386712620c6262e244b970802bf634ec9e81dc21"}
Jul 27 17:38:29.676274 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.676260    2564 scope.go:117] "RemoveContainer" containerID="f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"
Jul 27 17:38:29.676639 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:29.676613    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4\": container with ID starting with f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4 not found: ID does not exist" containerID="f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"
Jul 27 17:38:29.676731 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.676648    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4"} err="failed to get container status \"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4\": rpc error: code = NotFound desc = could not find container \"f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4\": container with ID starting with f9e6441389f091b75c24de957a5b59134118d00d7bc4e4b82dc28b76fd086ae4 not found: ID does not exist"
Jul 27 17:38:29.676731 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.676671    2564 scope.go:117] "RemoveContainer" containerID="144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"
Jul 27 17:38:29.685955 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.685938    2564 scope.go:117] "RemoveContainer" containerID="144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"
Jul 27 17:38:29.686205 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:29.686188    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e\": container with ID starting with 144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e not found: ID does not exist" containerID="144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"
Jul 27 17:38:29.686260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.686211    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e"} err="failed to get container status \"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e\": rpc error: code = NotFound desc = could not find container \"144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e\": container with ID starting with 144c8d75142475791a9688c87799f2cbbc1a1624209b1a1c8289f06ec5ec981e not found: ID does not exist"
Jul 27 17:38:29.686260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.686227    2564 scope.go:117] "RemoveContainer" containerID="7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"
Jul 27 17:38:29.692910 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.692891    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:38:29.693449 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.693433    2564 scope.go:117] "RemoveContainer" containerID="7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"
Jul 27 17:38:29.693675 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:29.693657    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07\": container with ID starting with 7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07 not found: ID does not exist" containerID="7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"
Jul 27 17:38:29.693761 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.693679    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07"} err="failed to get container status \"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07\": rpc error: code = NotFound desc = could not find container \"7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07\": container with ID starting with 7ab4b4b91be56611f53bd03c7f4037b718576f4db2ba23811856e868b205db07 not found: ID does not exist"
Jul 27 17:38:29.696698 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.696676    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-05e104-84f88dcdc6-z7vtk"]
Jul 27 17:38:29.707391 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.707371    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:38:29.713303 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.713285    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-05e104-79747d44bf-jr7hl"]
Jul 27 17:38:29.724218 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.724198    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:38:29.730341 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:29.730323    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-05e104-5d7f8c4ffb-s4gxv"]
Jul 27 17:38:30.659644 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:30.659603    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30400a06-d0dc-424d-aa74-e725e7948e1c" path="/var/lib/kubelet/pods/30400a06-d0dc-424d-aa74-e725e7948e1c/volumes"
Jul 27 17:38:30.660034 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:30.659993    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" path="/var/lib/kubelet/pods/6be63f3d-6e6b-44e6-9c14-4bb552572e48/volumes"
Jul 27 17:38:30.660293 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:30.660281    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88291ae4-4a7b-485e-a823-bf29573135ea" path="/var/lib/kubelet/pods/88291ae4-4a7b-485e-a823-bf29573135ea/volumes"
Jul 27 17:38:33.290496 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:33.290464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:38:44.139830 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.139802    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:38:44.140226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140210    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8740860d-308a-4b38-934e-20f782992095" containerName="payload-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140229    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8740860d-308a-4b38-934e-20f782992095" containerName="payload-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140239    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30400a06-d0dc-424d-aa74-e725e7948e1c" containerName="payload-pre-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140244    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="30400a06-d0dc-424d-aa74-e725e7948e1c" containerName="payload-pre-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140255    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88291ae4-4a7b-485e-a823-bf29573135ea" containerName="payload-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140260    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88291ae4-4a7b-485e-a823-bf29573135ea" containerName="payload-processing"
Jul 27 17:38:44.140271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140271    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140277    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140285    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140291    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140307    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67966780-c88f-4db9-9f80-ac5731be44be" containerName="payload-pre-processing"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140312    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67966780-c88f-4db9-9f80-ac5731be44be" containerName="payload-pre-processing"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140362    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8740860d-308a-4b38-934e-20f782992095" containerName="payload-processing"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140371    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="30400a06-d0dc-424d-aa74-e725e7948e1c" containerName="payload-pre-processing"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140378    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d5eec6-d58f-4b5c-9dc8-8b223a3221f2" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140384    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be63f3d-6e6b-44e6-9c14-4bb552572e48" containerName="maas-api"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140391    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88291ae4-4a7b-485e-a823-bf29573135ea" containerName="payload-processing"
Jul 27 17:38:44.140455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.140398    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67966780-c88f-4db9-9f80-ac5731be44be" containerName="payload-pre-processing"
Jul 27 17:38:44.145260 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.145241    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.150069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.150025    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-004618-serving-cert\""
Jul 27 17:38:44.150190 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.150077    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vcrrj\""
Jul 27 17:38:44.151666 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.151491    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 17:38:44.151666 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.151495    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 17:38:44.154248 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.154048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:38:44.224356 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.224329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.224475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.224388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fkd\" (UniqueName: \"kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.224522 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.224464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.325597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.325567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.325703 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.325640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.325703 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.325685    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fkd\" (UniqueName: \"kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.326375 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.326354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.328172 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.328148    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.339050 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.339025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fkd\" (UniqueName: \"kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd\") pod \"maas-api-e2e-shared-a-004618-66c7dc64c-f45gm\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.456174 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.456145    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:44.587529 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.587507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:38:44.589197 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:38:44.589168    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4bd945_4e03_4f68_a850_3d45b597001d.slice/crio-95f33d53ab9c5fd54ada16959d8178589f12f169fddc32b44e93d4170efe2883 WatchSource:0}: Error finding container 95f33d53ab9c5fd54ada16959d8178589f12f169fddc32b44e93d4170efe2883: Status 404 returned error can't find the container with id 95f33d53ab9c5fd54ada16959d8178589f12f169fddc32b44e93d4170efe2883
Jul 27 17:38:44.742740 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.742646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" event={"ID":"ea4bd945-4e03-4f68-a850-3d45b597001d","Type":"ContainerStarted","Data":"95f33d53ab9c5fd54ada16959d8178589f12f169fddc32b44e93d4170efe2883"}
Jul 27 17:38:44.899236 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.899203    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:38:44.903557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.903533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:44.907194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.907174    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-004618\""
Jul 27 17:38:44.907593 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.907576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-004618-dockercfg-8t2b6\""
Jul 27 17:38:44.922887 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.922861    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:38:44.930694 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.930673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpmf\" (UniqueName: \"kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:44.930791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.930720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:44.943021 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.942997    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:38:44.946124 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.946109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:44.958475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:44.958455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:38:45.031143 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.031081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpmf\" (UniqueName: \"kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.031143 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.031112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.031143 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.031136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2q22\" (UniqueName: \"kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.031394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.031176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.031873 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.031855    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.041710 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.041688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpmf\" (UniqueName: \"kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf\") pod \"payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") " pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.131869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.131831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.131869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.131866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2q22\" (UniqueName: \"kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.132473 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.132453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.152853 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.152830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2q22\" (UniqueName: \"kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22\") pod \"payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.214429 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.214409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.255093 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.255066    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.362272 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.362221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:38:45.364234 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:38:45.364194    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6e78f4_f2df_43b5_874a_0095dd4a1f17.slice/crio-8f548a0e016cf981c0320e481e606a57a090e3a15947308ed84ef5762cf8c039 WatchSource:0}: Error finding container 8f548a0e016cf981c0320e481e606a57a090e3a15947308ed84ef5762cf8c039: Status 404 returned error can't find the container with id 8f548a0e016cf981c0320e481e606a57a090e3a15947308ed84ef5762cf8c039
Jul 27 17:38:45.409683 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.409658    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:38:45.410408 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:38:45.410382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7dfe4c_4ecb_444f_950e_53ed7b72917b.slice/crio-6ce0708698834472e94cf749c3f7b8d3d7ee77c3cda211f1f58fd66738979e97 WatchSource:0}: Error finding container 6ce0708698834472e94cf749c3f7b8d3d7ee77c3cda211f1f58fd66738979e97: Status 404 returned error can't find the container with id 6ce0708698834472e94cf749c3f7b8d3d7ee77c3cda211f1f58fd66738979e97
Jul 27 17:38:45.749868 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.749836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" event={"ID":"ea4bd945-4e03-4f68-a850-3d45b597001d","Type":"ContainerStarted","Data":"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"}
Jul 27 17:38:45.750054 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.749905    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:45.751347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.751318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" event={"ID":"3f7dfe4c-4ecb-444f-950e-53ed7b72917b","Type":"ContainerStarted","Data":"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"}
Jul 27 17:38:45.751483 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.751375    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:38:45.751483 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.751397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" event={"ID":"3f7dfe4c-4ecb-444f-950e-53ed7b72917b","Type":"ContainerStarted","Data":"6ce0708698834472e94cf749c3f7b8d3d7ee77c3cda211f1f58fd66738979e97"}
Jul 27 17:38:45.752869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.752847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" event={"ID":"4f6e78f4-f2df-43b5-874a-0095dd4a1f17","Type":"ContainerStarted","Data":"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"}
Jul 27 17:38:45.752973 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.752901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" event={"ID":"4f6e78f4-f2df-43b5-874a-0095dd4a1f17","Type":"ContainerStarted","Data":"8f548a0e016cf981c0320e481e606a57a090e3a15947308ed84ef5762cf8c039"}
Jul 27 17:38:45.753039 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.752994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:45.774912 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.772861    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" podStartSLOduration=0.771009175 podStartE2EDuration="1.7728443s" totalImagesPullingTime="1.001835125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:45.771331228 +0000 UTC m=+2545.603696204" watchObservedRunningTime="2026-07-27 17:38:45.7728443 +0000 UTC m=+2545.605209278"
Jul 27 17:38:45.797271 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.797230    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" podStartSLOduration=1.7972166330000001 podStartE2EDuration="1.797216633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:45.795500547 +0000 UTC m=+2545.627865583" watchObservedRunningTime="2026-07-27 17:38:45.797216633 +0000 UTC m=+2545.629581609"
Jul 27 17:38:45.827788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:45.827749    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" podStartSLOduration=1.82773201 podStartE2EDuration="1.82773201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:45.827356008 +0000 UTC m=+2545.659720984" watchObservedRunningTime="2026-07-27 17:38:45.82773201 +0000 UTC m=+2545.660096987"
Jul 27 17:38:51.500484 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:51.500447    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:38:51.762553 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:51.762490    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:38:52.213854 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.213824    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:38:52.217554 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.217532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.220901 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.220860    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-509a5d-serving-cert\""
Jul 27 17:38:52.233845 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.233819    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:38:52.294935 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.294910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.295026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.294975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.295026 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.295002    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8kg\" (UniqueName: \"kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.395755 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.395727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.395859 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.395806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.395859 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.395837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8kg\" (UniqueName: \"kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.396015 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:52.395858    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-serving-cert: secret "maas-api-e2e-shared-b-509a5d-serving-cert" not found
Jul 27 17:38:52.396015 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:38:52.395937    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls podName:1f19d9a6-2727-4d47-b4f9-bc5de4a52a04 nodeName:}" failed. No retries permitted until 2026-07-27 17:38:52.895919616 +0000 UTC m=+2552.728284571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls") pod "maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" (UID: "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04") : secret "maas-api-e2e-shared-b-509a5d-serving-cert" not found
Jul 27 17:38:52.396411 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.396394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.411238 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.411213    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8kg\" (UniqueName: \"kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.899654 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.899623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.902022 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.901997    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") pod \"maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:52.965176 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.965150    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:38:52.968657 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.968640    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:52.972490 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.972467    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-509a5d\""
Jul 27 17:38:52.972647 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.972630    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-509a5d-dockercfg-vgvtc\""
Jul 27 17:38:52.980081 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:52.980059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:38:53.100727 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.100704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88m8s\" (UniqueName: \"kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.100866 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.100747    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.127367 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.127342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:53.201461 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.201434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.201606 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.201536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88m8s\" (UniqueName: \"kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.202359 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.202334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.210439 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.210413    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88m8s\" (UniqueName: \"kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s\") pod \"payload-processing-e2e-shared-b-509a5d-745f757454-hg644\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.282196 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.282161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.409982 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.409845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:38:53.412785 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:38:53.412762    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107421f9_d5a2_4332_bd03_2f01a391dea6.slice/crio-7814fb0e9e314164f22dd70572f44566963eebc342cae5eeb62567d85404e83c WatchSource:0}: Error finding container 7814fb0e9e314164f22dd70572f44566963eebc342cae5eeb62567d85404e83c: Status 404 returned error can't find the container with id 7814fb0e9e314164f22dd70572f44566963eebc342cae5eeb62567d85404e83c
Jul 27 17:38:53.462258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.462238    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:38:53.467925 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:38:53.467897    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f19d9a6_2727_4d47_b4f9_bc5de4a52a04.slice/crio-03a398f41ebd5f40bb5ba208f66979d32aa518809d50a1d01f9c7b4084d22acf WatchSource:0}: Error finding container 03a398f41ebd5f40bb5ba208f66979d32aa518809d50a1d01f9c7b4084d22acf: Status 404 returned error can't find the container with id 03a398f41ebd5f40bb5ba208f66979d32aa518809d50a1d01f9c7b4084d22acf
Jul 27 17:38:53.789983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.789900    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" event={"ID":"107421f9-d5a2-4332-bd03-2f01a391dea6","Type":"ContainerStarted","Data":"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"}
Jul 27 17:38:53.789983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.789957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" event={"ID":"107421f9-d5a2-4332-bd03-2f01a391dea6","Type":"ContainerStarted","Data":"7814fb0e9e314164f22dd70572f44566963eebc342cae5eeb62567d85404e83c"}
Jul 27 17:38:53.790217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.790008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:38:53.791182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.791159    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" event={"ID":"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04","Type":"ContainerStarted","Data":"03a398f41ebd5f40bb5ba208f66979d32aa518809d50a1d01f9c7b4084d22acf"}
Jul 27 17:38:53.808713 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:53.808673    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" podStartSLOduration=1.808657138 podStartE2EDuration="1.808657138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:53.808209001 +0000 UTC m=+2553.640574001" watchObservedRunningTime="2026-07-27 17:38:53.808657138 +0000 UTC m=+2553.641022114"
Jul 27 17:38:54.797078 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:54.797043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" event={"ID":"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04","Type":"ContainerStarted","Data":"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"}
Jul 27 17:38:54.817327 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:54.817283    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" podStartSLOduration=2.394116402 podStartE2EDuration="2.817269168s" totalImagesPullingTime="423.152766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:54.815621033 +0000 UTC m=+2554.647985987" watchObservedRunningTime="2026-07-27 17:38:54.817269168 +0000 UTC m=+2554.649634146"
Jul 27 17:38:55.801457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:55.801427    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:38:56.759359 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:56.759326    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:38:56.759566 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:38:56.759399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:39:01.811209 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:01.811181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:39:02.279400 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:02.279371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:39:04.799895 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:04.799851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:39:07.638042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.638006    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:39:07.641527 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.641508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.646311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.646227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"default-dockercfg-fqjh4\""
Jul 27 17:39:07.646311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.646239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"kube-root-ca.crt\""
Jul 27 17:39:07.646311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.646256    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"openshift-service-ca.crt\""
Jul 27 17:39:07.646311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.646305    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"auth-test-model-004618-kserve-self-signed-certs\""
Jul 27 17:39:07.653710 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.653691    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:39:07.719568 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.719665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.719665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719653    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.719757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqd6v\" (UniqueName: \"kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.719757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.719757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.719734    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821061 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821039    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqd6v\" (UniqueName: \"kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821416 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821480 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821539    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.821597 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.821574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.823247 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.823227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.823439 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.823423    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.830068 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.830041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqd6v\" (UniqueName: \"kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v\") pod \"auth-test-model-004618-kserve-c87766667-8nckf\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") " pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:07.952993 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:07.952969    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:08.085609 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:08.085581    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:39:08.086299 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:39:08.086269    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f733801_e9a6_4b0d_84b0_35f91087e8f3.slice/crio-b2557430865edfe8eca04b3ddec0f3653d7265c3f5028df8d490c7b93a413b5a WatchSource:0}: Error finding container b2557430865edfe8eca04b3ddec0f3653d7265c3f5028df8d490c7b93a413b5a: Status 404 returned error can't find the container with id b2557430865edfe8eca04b3ddec0f3653d7265c3f5028df8d490c7b93a413b5a
Jul 27 17:39:08.858924 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:08.858869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" event={"ID":"7f733801-e9a6-4b0d-84b0-35f91087e8f3","Type":"ContainerStarted","Data":"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"}
Jul 27 17:39:08.858924 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:08.858926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" event={"ID":"7f733801-e9a6-4b0d-84b0-35f91087e8f3","Type":"ContainerStarted","Data":"b2557430865edfe8eca04b3ddec0f3653d7265c3f5028df8d490c7b93a413b5a"}
Jul 27 17:39:08.880953 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:08.880906    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" podStartSLOduration=1.880891678 podStartE2EDuration="1.880891678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:08.878507097 +0000 UTC m=+2568.710872085" watchObservedRunningTime="2026-07-27 17:39:08.880891678 +0000 UTC m=+2568.713256644"
Jul 27 17:39:13.089831 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:13.089801    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:39:17.953482 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:17.953453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:17.953970 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:17.953499    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:17.965804 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:17.965784    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:18.919352 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:18.919321    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:39:34.110240 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.110207    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:39:34.113658 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.113638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.118135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.118113    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"kube-root-ca.crt\""
Jul 27 17:39:34.118135 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.118124    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"default-dockercfg-6j2pn\""
Jul 27 17:39:34.118324 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.118114    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"openshift-service-ca.crt\""
Jul 27 17:39:34.118324 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.118151    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"auth-test-model-509a5d-kserve-self-signed-certs\""
Jul 27 17:39:34.124742 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.124722    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:39:34.129631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.129751 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbn2\" (UniqueName: \"kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.129751 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129659    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.129751 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.129907 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.129907 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.129822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230355 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230486 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230365    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230486 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230486 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230654 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbn2\" (UniqueName: \"kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230654 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230760 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230817 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.230871 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.230830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.232845 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.232820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.232999 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.232979    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.239525 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.239507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbn2\" (UniqueName: \"kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2\") pod \"auth-test-model-509a5d-kserve-569f9db64c-s7l67\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") " pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.423849 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.423819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:34.554608 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.554584    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:39:34.556712 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:39:34.556687    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f76769_1470_49c5_86c6_33b9ace40289.slice/crio-2a7f0d8b56f22a8ebbdb73c97216427bd2adcfb95b9dcaec5e9b24d38d73a572 WatchSource:0}: Error finding container 2a7f0d8b56f22a8ebbdb73c97216427bd2adcfb95b9dcaec5e9b24d38d73a572: Status 404 returned error can't find the container with id 2a7f0d8b56f22a8ebbdb73c97216427bd2adcfb95b9dcaec5e9b24d38d73a572
Jul 27 17:39:34.985859 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.985824    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" event={"ID":"e1f76769-1470-49c5-86c6-33b9ace40289","Type":"ContainerStarted","Data":"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"}
Jul 27 17:39:34.986044 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:34.985872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" event={"ID":"e1f76769-1470-49c5-86c6-33b9ace40289","Type":"ContainerStarted","Data":"2a7f0d8b56f22a8ebbdb73c97216427bd2adcfb95b9dcaec5e9b24d38d73a572"}
Jul 27 17:39:35.005442 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:35.005395    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" podStartSLOduration=1.005380289 podStartE2EDuration="1.005380289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:35.003215324 +0000 UTC m=+2594.835580299" watchObservedRunningTime="2026-07-27 17:39:35.005380289 +0000 UTC m=+2594.837745266"
Jul 27 17:39:37.796351 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:37.796323    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:39:44.424537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:44.424453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:44.424537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:44.424504    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:44.438736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:44.438716    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:39:45.043639 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:39:45.043599    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:40:03.290825 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:40:03.290792    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:40:24.590386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:40:24.590354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:40:42.689093 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:40:42.689058    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:41:10.178374 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:41:10.178343    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:41:20.778189 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:41:20.778161    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:41:20.783801 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:41:20.783776    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:41:25.600432 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:41:25.600395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:41:45.884146 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:41:45.884116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:42:03.900977 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:42:03.899334    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:42:26.884042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:42:26.884007    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:42:44.989253 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:42:44.989171    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:43:06.187894 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:06.187859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:43:23.994872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:23.994834    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:43:44.580250 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:44.580216    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:43:49.723594 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:49.723560    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:43:50.613493 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:50.613462    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:43:53.008156 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.008120    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:43:53.011648 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.011626    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.014383 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.014363    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"sub-test-model-004618-kserve-self-signed-certs\""
Jul 27 17:43:53.022905 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.022865    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:43:53.102763 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.102873 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.102873 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102802    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkg2\" (UniqueName: \"kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.102873 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.103024 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.103024 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.102931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204041 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204041 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204197 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204189    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkg2\" (UniqueName: \"kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204509 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204490    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204584 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.204665 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.204574    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.206243 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.206219    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.206556 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.206536    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.212920 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.212868    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkg2\" (UniqueName: \"kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2\") pod \"sub-test-model-004618-kserve-69b8dc4849-kz9tc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") " pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.320690 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.320634    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:43:53.454083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:53.454054    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:43:53.455852 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:43:53.455827    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod351729b3_7cb8_4e3e_8d53_28442ecdb4cc.slice/crio-2250f48dabb9f1a3a7c007073daa97e4912b01a009f3d4a1c43566d0af42bf21 WatchSource:0}: Error finding container 2250f48dabb9f1a3a7c007073daa97e4912b01a009f3d4a1c43566d0af42bf21: Status 404 returned error can't find the container with id 2250f48dabb9f1a3a7c007073daa97e4912b01a009f3d4a1c43566d0af42bf21
Jul 27 17:43:54.183286 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:54.183250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" event={"ID":"351729b3-7cb8-4e3e-8d53-28442ecdb4cc","Type":"ContainerStarted","Data":"b1fb01f1921d1977b59935002fb73ba20ef8873a9a0166421cf5e61635e01611"}
Jul 27 17:43:54.183286 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:54.183289    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" event={"ID":"351729b3-7cb8-4e3e-8d53-28442ecdb4cc","Type":"ContainerStarted","Data":"2250f48dabb9f1a3a7c007073daa97e4912b01a009f3d4a1c43566d0af42bf21"}
Jul 27 17:43:54.204950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:43:54.204905    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" podStartSLOduration=2.204887945 podStartE2EDuration="2.204887945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:43:54.203085326 +0000 UTC m=+2854.035450314" watchObservedRunningTime="2026-07-27 17:43:54.204887945 +0000 UTC m=+2854.037252914"
Jul 27 17:44:01.278139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:01.278099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:44:03.321405 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:03.321374    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:44:03.321855 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:03.321421    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:44:03.334028 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:03.334008    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:44:04.242302 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:04.242270    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:44:04.742552 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:04.742511    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" podUID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" containerName="main" containerID="cri-o://28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd" gracePeriod=45
Jul 27 17:44:04.987065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:04.987043    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:44:05.104401 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104333    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104401 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104379    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104406    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqd6v\" (UniqueName: \"kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104447    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104488    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs\") pod \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\" (UID: \"7f733801-e9a6-4b0d-84b0-35f91087e8f3\") "
Jul 27 17:44:05.104805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104615    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.104805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home" (OuterVolumeSpecName: "home") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.104805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.104996 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104817    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.104996 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104829    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.104996 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.104838    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.106757 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.106716    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:44:05.106855 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.106753    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm" (OuterVolumeSpecName: "dshm") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.106855 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.106806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v" (OuterVolumeSpecName: "kube-api-access-gqd6v") pod "7f733801-e9a6-4b0d-84b0-35f91087e8f3" (UID: "7f733801-e9a6-4b0d-84b0-35f91087e8f3"). InnerVolumeSpecName "kube-api-access-gqd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:44:05.206231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.206211    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqd6v\" (UniqueName: \"kubernetes.io/projected/7f733801-e9a6-4b0d-84b0-35f91087e8f3-kube-api-access-gqd6v\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.206231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.206230    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f733801-e9a6-4b0d-84b0-35f91087e8f3-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.206366 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.206240    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f733801-e9a6-4b0d-84b0-35f91087e8f3-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.236868 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.236840    2564 generic.go:356] "Generic (PLEG): container finished" podID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" containerID="28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd" exitCode=0
Jul 27 17:44:05.236979 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.236896    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"
Jul 27 17:44:05.236979 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.236930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" event={"ID":"7f733801-e9a6-4b0d-84b0-35f91087e8f3","Type":"ContainerDied","Data":"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"}
Jul 27 17:44:05.236979 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.236972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf" event={"ID":"7f733801-e9a6-4b0d-84b0-35f91087e8f3","Type":"ContainerDied","Data":"b2557430865edfe8eca04b3ddec0f3653d7265c3f5028df8d490c7b93a413b5a"}
Jul 27 17:44:05.237116 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.236990    2564 scope.go:117] "RemoveContainer" containerID="28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"
Jul 27 17:44:05.246040 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.246025    2564 scope.go:117] "RemoveContainer" containerID="28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"
Jul 27 17:44:05.246295 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:44:05.246278    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd\": container with ID starting with 28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd not found: ID does not exist" containerID="28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"
Jul 27 17:44:05.246345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.246302    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd"} err="failed to get container status \"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd\": rpc error: code = NotFound desc = could not find container \"28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd\": container with ID starting with 28731813780bb7786f16f51bf782009c78861e395195b91d4580097f1c9bb9cd not found: ID does not exist"
Jul 27 17:44:05.258986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.258967    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:44:05.263052 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.263034    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-004618/auth-test-model-004618-kserve-c87766667-8nckf"]
Jul 27 17:44:05.632014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.631985    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" podUID="e1f76769-1470-49c5-86c6-33b9ace40289" containerName="main" containerID="cri-o://3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219" gracePeriod=45
Jul 27 17:44:05.872526 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.872499    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:44:05.911141 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911122    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911249 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911165    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911249 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911182    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911249 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911240    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcbn2\" (UniqueName: \"kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911415 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911271    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911415 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911302    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir\") pod \"e1f76769-1470-49c5-86c6-33b9ace40289\" (UID: \"e1f76769-1470-49c5-86c6-33b9ace40289\") "
Jul 27 17:44:05.911415 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.911573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911549    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home" (OuterVolumeSpecName: "home") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.911573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911566    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.911687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911641    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.911687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911660    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.911687 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.911675    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:05.913296 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.913278    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm" (OuterVolumeSpecName: "dshm") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:05.913656 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.913633    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:44:05.913656 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:05.913649    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2" (OuterVolumeSpecName: "kube-api-access-jcbn2") pod "e1f76769-1470-49c5-86c6-33b9ace40289" (UID: "e1f76769-1470-49c5-86c6-33b9ace40289"). InnerVolumeSpecName "kube-api-access-jcbn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:44:06.012203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.012175    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f76769-1470-49c5-86c6-33b9ace40289-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:06.012203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.012201    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbn2\" (UniqueName: \"kubernetes.io/projected/e1f76769-1470-49c5-86c6-33b9ace40289-kube-api-access-jcbn2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:06.012358 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.012217    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f76769-1470-49c5-86c6-33b9ace40289-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:44:06.242709 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.242630    2564 generic.go:356] "Generic (PLEG): container finished" podID="e1f76769-1470-49c5-86c6-33b9ace40289" containerID="3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219" exitCode=0
Jul 27 17:44:06.242709 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.242672    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"
Jul 27 17:44:06.242938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.242718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" event={"ID":"e1f76769-1470-49c5-86c6-33b9ace40289","Type":"ContainerDied","Data":"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"}
Jul 27 17:44:06.242938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.242757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67" event={"ID":"e1f76769-1470-49c5-86c6-33b9ace40289","Type":"ContainerDied","Data":"2a7f0d8b56f22a8ebbdb73c97216427bd2adcfb95b9dcaec5e9b24d38d73a572"}
Jul 27 17:44:06.242938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.242779    2564 scope.go:117] "RemoveContainer" containerID="3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"
Jul 27 17:44:06.251240 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.251222    2564 scope.go:117] "RemoveContainer" containerID="3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"
Jul 27 17:44:06.251502 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:44:06.251477    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219\": container with ID starting with 3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219 not found: ID does not exist" containerID="3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"
Jul 27 17:44:06.251549 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.251510    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219"} err="failed to get container status \"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219\": rpc error: code = NotFound desc = could not find container \"3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219\": container with ID starting with 3ef62d19e05e817f4d5ff9d4e8fa0a77bb183d9573339ea07ce9c90bfdee7219 not found: ID does not exist"
Jul 27 17:44:06.265299 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.265277    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:44:06.269798 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.269780    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/auth-test-model-509a5d-kserve-569f9db64c-s7l67"]
Jul 27 17:44:06.659370 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.659342    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" path="/var/lib/kubelet/pods/7f733801-e9a6-4b0d-84b0-35f91087e8f3/volumes"
Jul 27 17:44:06.659723 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:06.659709    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f76769-1470-49c5-86c6-33b9ace40289" path="/var/lib/kubelet/pods/e1f76769-1470-49c5-86c6-33b9ace40289/volumes"
Jul 27 17:44:12.808646 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.808433    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809671    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" containerName="main"
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809688    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" containerName="main"
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809727    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f76769-1470-49c5-86c6-33b9ace40289" containerName="main"
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809734    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f76769-1470-49c5-86c6-33b9ace40289" containerName="main"
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809864    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f733801-e9a6-4b0d-84b0-35f91087e8f3" containerName="main"
Jul 27 17:44:12.809898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.809897    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f76769-1470-49c5-86c6-33b9ace40289" containerName="main"
Jul 27 17:44:12.814511 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.814486    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.817903 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.817857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"openshift-service-ca.crt\""
Jul 27 17:44:12.819464 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.819443    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"kube-root-ca.crt\""
Jul 27 17:44:12.819662 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.819446    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"default-dockercfg-6j2pn\""
Jul 27 17:44:12.819830 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.819472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"sub-test-model-509a5d-kserve-self-signed-certs\""
Jul 27 17:44:12.821929 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.821906    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:44:12.855309 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.855434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855339    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.855434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.855434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9l9\" (UniqueName: \"kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.855434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.855434 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.855434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955795 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9l9\" (UniqueName: \"kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.955975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.955934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.956286 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.956262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.956338 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.956302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.956383 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.956374    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.958023 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.958002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.958259 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.958245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:12.964531 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:12.964513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9l9\" (UniqueName: \"kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9\") pod \"sub-test-model-509a5d-kserve-59ddb44459-jxbht\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") " pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:13.125703 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:13.125625    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:13.258184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:13.258158    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:44:13.259842 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:44:13.259813    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28123198_1f70_4364_b20c_e89411cf3873.slice/crio-f0b30d119ba4fac0cc156269addb5056ed6a7b57faf552cdb0a6fb9f1da45126 WatchSource:0}: Error finding container f0b30d119ba4fac0cc156269addb5056ed6a7b57faf552cdb0a6fb9f1da45126: Status 404 returned error can't find the container with id f0b30d119ba4fac0cc156269addb5056ed6a7b57faf552cdb0a6fb9f1da45126
Jul 27 17:44:13.279658 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:13.279634    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" event={"ID":"28123198-1f70-4364-b20c-e89411cf3873","Type":"ContainerStarted","Data":"f0b30d119ba4fac0cc156269addb5056ed6a7b57faf552cdb0a6fb9f1da45126"}
Jul 27 17:44:14.285226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:14.285191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" event={"ID":"28123198-1f70-4364-b20c-e89411cf3873","Type":"ContainerStarted","Data":"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"}
Jul 27 17:44:14.315342 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:14.315293    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" podStartSLOduration=2.315277925 podStartE2EDuration="2.315277925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:44:14.313609981 +0000 UTC m=+2874.145974956" watchObservedRunningTime="2026-07-27 17:44:14.315277925 +0000 UTC m=+2874.147642901"
Jul 27 17:44:15.989357 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:15.989318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:44:23.126485 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:23.126446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:23.126872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:23.126507    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:23.139669 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:23.139646    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:44:23.337409 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:44:23.337384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:45:15.385505 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:45:15.385471    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:45:31.992388 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:45:31.992354    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:45:45.541501 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:45:45.541428    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:45:46.439194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:45:46.439162    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:45:47.796760 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:45:47.796727    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:46:00.540562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.540525    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:46:00.544001 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.543981    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.547294 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.547270    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-004618\"/\"rate-test-model-004618-kserve-self-signed-certs\""
Jul 27 17:46:00.555085 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.555061    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:46:00.559720 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.559690    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" podUID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" containerName="main" containerID="cri-o://b1fb01f1921d1977b59935002fb73ba20ef8873a9a0166421cf5e61635e01611" gracePeriod=45
Jul 27 17:46:00.628676 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628641    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.628792 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.628792 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.628898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.628898 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628862    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.628988 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.628907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcz4\" (UniqueName: \"kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.698996 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.698968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:46:00.729471 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729624 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729624 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729624 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729808 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729808 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcz4\" (UniqueName: \"kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.729941 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.729832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.730423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.730300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.730423 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.730368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.732329 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.732309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.732567 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.732550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.740121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.740097    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcz4\" (UniqueName: \"kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4\") pod \"rate-test-model-004618-kserve-79bbf4d5cf-s4ww5\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") " pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.773705 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.773680    2564 generic.go:356] "Generic (PLEG): container finished" podID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" containerID="b1fb01f1921d1977b59935002fb73ba20ef8873a9a0166421cf5e61635e01611" exitCode=0
Jul 27 17:46:00.773820 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.773759    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" event={"ID":"351729b3-7cb8-4e3e-8d53-28442ecdb4cc","Type":"ContainerDied","Data":"b1fb01f1921d1977b59935002fb73ba20ef8873a9a0166421cf5e61635e01611"}
Jul 27 17:46:00.815890 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.815857    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:46:00.853429 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.853404    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:00.931768 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.931737    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.931937 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.931852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdkg2\" (UniqueName: \"kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.932010 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.931941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.932010 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.931990    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.932116 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932024    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.932116 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932048    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm\") pod \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\" (UID: \"351729b3-7cb8-4e3e-8d53-28442ecdb4cc\") "
Jul 27 17:46:00.932211 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932178    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:00.932498 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932385    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:00.932596 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932500    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home" (OuterVolumeSpecName: "home") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:00.932596 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.932554    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:00.935394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.934149    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm" (OuterVolumeSpecName: "dshm") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:00.935394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.934996    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2" (OuterVolumeSpecName: "kube-api-access-bdkg2") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "kube-api-access-bdkg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:46:00.935394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.935325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "351729b3-7cb8-4e3e-8d53-28442ecdb4cc" (UID: "351729b3-7cb8-4e3e-8d53-28442ecdb4cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:46:00.988267 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:00.988241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:46:00.989714 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:46:00.989692    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb25803_320e_4788_9967_c5970c84de80.slice/crio-861a6cfb87f2cc984274cc054fd0ca702e8b70a586a345b04ab72ed81a6b729a WatchSource:0}: Error finding container 861a6cfb87f2cc984274cc054fd0ca702e8b70a586a345b04ab72ed81a6b729a: Status 404 returned error can't find the container with id 861a6cfb87f2cc984274cc054fd0ca702e8b70a586a345b04ab72ed81a6b729a
Jul 27 17:46:01.033412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.033391    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdkg2\" (UniqueName: \"kubernetes.io/projected/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-kube-api-access-bdkg2\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.033412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.033412    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.033584 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.033422    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.033584 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.033430    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.033584 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.033437    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/351729b3-7cb8-4e3e-8d53-28442ecdb4cc-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.457583 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.457539    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" podUID="28123198-1f70-4364-b20c-e89411cf3873" containerName="main" containerID="cri-o://d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59" gracePeriod=45
Jul 27 17:46:01.708311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.708258    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:46:01.779050 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.779022    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"
Jul 27 17:46:01.779174 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.779023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc" event={"ID":"351729b3-7cb8-4e3e-8d53-28442ecdb4cc","Type":"ContainerDied","Data":"2250f48dabb9f1a3a7c007073daa97e4912b01a009f3d4a1c43566d0af42bf21"}
Jul 27 17:46:01.779174 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.779135    2564 scope.go:117] "RemoveContainer" containerID="b1fb01f1921d1977b59935002fb73ba20ef8873a9a0166421cf5e61635e01611"
Jul 27 17:46:01.780734 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.780710    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" event={"ID":"8bb25803-320e-4788-9967-c5970c84de80","Type":"ContainerStarted","Data":"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"}
Jul 27 17:46:01.780848 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.780743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" event={"ID":"8bb25803-320e-4788-9967-c5970c84de80","Type":"ContainerStarted","Data":"861a6cfb87f2cc984274cc054fd0ca702e8b70a586a345b04ab72ed81a6b729a"}
Jul 27 17:46:01.781988 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.781963    2564 generic.go:356] "Generic (PLEG): container finished" podID="28123198-1f70-4364-b20c-e89411cf3873" containerID="d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59" exitCode=0
Jul 27 17:46:01.782102 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.782004    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"
Jul 27 17:46:01.782102 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.782035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" event={"ID":"28123198-1f70-4364-b20c-e89411cf3873","Type":"ContainerDied","Data":"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"}
Jul 27 17:46:01.782102 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.782060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht" event={"ID":"28123198-1f70-4364-b20c-e89411cf3873","Type":"ContainerDied","Data":"f0b30d119ba4fac0cc156269addb5056ed6a7b57faf552cdb0a6fb9f1da45126"}
Jul 27 17:46:01.787731 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.787713    2564 scope.go:117] "RemoveContainer" containerID="d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"
Jul 27 17:46:01.795031 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.795015    2564 scope.go:117] "RemoveContainer" containerID="d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"
Jul 27 17:46:01.795283 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:46:01.795264    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59\": container with ID starting with d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59 not found: ID does not exist" containerID="d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"
Jul 27 17:46:01.795346 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.795294    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59"} err="failed to get container status \"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59\": rpc error: code = NotFound desc = could not find container \"d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59\": container with ID starting with d0486db0c99a72904c32dbbcf37be576ba2f7f44fba5a0d707134aeab2354e59 not found: ID does not exist"
Jul 27 17:46:01.815993 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.815958    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" podStartSLOduration=1.815945287 podStartE2EDuration="1.815945287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:01.814175194 +0000 UTC m=+2981.646540170" watchObservedRunningTime="2026-07-27 17:46:01.815945287 +0000 UTC m=+2981.648310241"
Jul 27 17:46:01.833720 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.833700    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:46:01.839182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839158    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-004618/sub-test-model-004618-kserve-69b8dc4849-kz9tc"]
Jul 27 17:46:01.839828 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839813    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.839889 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.839944 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839932    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz9l9\" (UniqueName: \"kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.839983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.839983 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.839973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.840082 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.840026    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm\") pod \"28123198-1f70-4364-b20c-e89411cf3873\" (UID: \"28123198-1f70-4364-b20c-e89411cf3873\") "
Jul 27 17:46:01.840141 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.840090    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache" (OuterVolumeSpecName: "model-cache") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:01.840298 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.840279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:01.840298 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.840267    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home" (OuterVolumeSpecName: "home") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:01.840437 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.840342    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.842227 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.842202    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm" (OuterVolumeSpecName: "dshm") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:01.842227 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.842206    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9" (OuterVolumeSpecName: "kube-api-access-mz9l9") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "kube-api-access-mz9l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:46:01.842392 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.842294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28123198-1f70-4364-b20c-e89411cf3873" (UID: "28123198-1f70-4364-b20c-e89411cf3873"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:46:01.941577 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.941554    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28123198-1f70-4364-b20c-e89411cf3873-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.941577 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.941576    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz9l9\" (UniqueName: \"kubernetes.io/projected/28123198-1f70-4364-b20c-e89411cf3873-kube-api-access-mz9l9\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.941696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.941585    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.941696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.941593    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:01.941696 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:01.941601    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28123198-1f70-4364-b20c-e89411cf3873-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:46:02.104443 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:02.104420    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:46:02.106799 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:02.106778    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/sub-test-model-509a5d-kserve-59ddb44459-jxbht"]
Jul 27 17:46:02.658333 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:02.658302    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28123198-1f70-4364-b20c-e89411cf3873" path="/var/lib/kubelet/pods/28123198-1f70-4364-b20c-e89411cf3873/volumes"
Jul 27 17:46:02.658686 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:02.658672    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" path="/var/lib/kubelet/pods/351729b3-7cb8-4e3e-8d53-28442ecdb4cc/volumes"
Jul 27 17:46:09.985401 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:09.985362    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:46:10.854431 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:10.854395    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:10.854598 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:10.854453    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:10.866663 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:10.866642    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:11.842984 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:11.842952    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:46:20.806917 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:20.806888    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:46:20.818018 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:20.817991    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:46:28.109697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.109667    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:46:28.110096 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110058    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28123198-1f70-4364-b20c-e89411cf3873" containerName="main"
Jul 27 17:46:28.110096 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110072    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28123198-1f70-4364-b20c-e89411cf3873" containerName="main"
Jul 27 17:46:28.110096 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110081    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" containerName="main"
Jul 27 17:46:28.110096 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110087    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" containerName="main"
Jul 27 17:46:28.110237 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110162    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28123198-1f70-4364-b20c-e89411cf3873" containerName="main"
Jul 27 17:46:28.110237 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.110171    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="351729b3-7cb8-4e3e-8d53-28442ecdb4cc" containerName="main"
Jul 27 17:46:28.114725 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.114705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.119378 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.119360    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"rate-test-model-509a5d-kserve-self-signed-certs\""
Jul 27 17:46:28.119487 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.119407    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"default-dockercfg-6j2pn\""
Jul 27 17:46:28.119562 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.119486    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"openshift-service-ca.crt\""
Jul 27 17:46:28.119631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.119597    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-509a5d\"/\"kube-root-ca.crt\""
Jul 27 17:46:28.123286 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.123265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:46:28.148128 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.148234 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148136    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.148234 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.148339 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.148339 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbdf\" (UniqueName: \"kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.148339 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.148277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249557 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbdf\" (UniqueName: \"kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.249697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.250015 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.249991    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.250015 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.250010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.250134 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.250109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.251990 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.251970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.252149 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.252131    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.261652 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.261633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbdf\" (UniqueName: \"kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf\") pod \"rate-test-model-509a5d-kserve-787d56c755-gq4jn\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") " pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.424791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.424760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:28.556492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.556468    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:46:28.558588 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:46:28.558554    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddacc3ddf_fedb_49f8_a323_992d4f0c82f4.slice/crio-29c0e013f0ee288e4471fd80b857904775654c27efbd95701ecf52f03d622e60 WatchSource:0}: Error finding container 29c0e013f0ee288e4471fd80b857904775654c27efbd95701ecf52f03d622e60: Status 404 returned error can't find the container with id 29c0e013f0ee288e4471fd80b857904775654c27efbd95701ecf52f03d622e60
Jul 27 17:46:28.907444 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.907411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" event={"ID":"dacc3ddf-fedb-49f8-a323-992d4f0c82f4","Type":"ContainerStarted","Data":"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"}
Jul 27 17:46:28.907628 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.907456    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" event={"ID":"dacc3ddf-fedb-49f8-a323-992d4f0c82f4","Type":"ContainerStarted","Data":"29c0e013f0ee288e4471fd80b857904775654c27efbd95701ecf52f03d622e60"}
Jul 27 17:46:28.927120 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:28.927071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" podStartSLOduration=0.927056642 podStartE2EDuration="927.056642ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:28.925802549 +0000 UTC m=+3008.758167524" watchObservedRunningTime="2026-07-27 17:46:28.927056642 +0000 UTC m=+3008.759421618"
Jul 27 17:46:31.786938 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:31.786904    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:46:38.425768 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:38.425727    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:38.425768 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:38.425780    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:38.438558 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:38.438532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:46:38.966595 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:46:38.966570    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:47:15.388961 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:15.388923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:47:36.397958 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:36.397925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:47:42.941767 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:42.941734    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:47:43.723583 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:43.723546    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:47:55.490231 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:55.490197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:47:57.960924 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:57.960863    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" podUID="8bb25803-320e-4788-9967-c5970c84de80" containerName="main" containerID="cri-o://509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c" gracePeriod=45
Jul 27 17:47:58.209916 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.209889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:47:58.277451 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277383    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277451 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fcz4\" (UniqueName: \"kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277482    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277500    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277627 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277593    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir\") pod \"8bb25803-320e-4788-9967-c5970c84de80\" (UID: \"8bb25803-320e-4788-9967-c5970c84de80\") "
Jul 27 17:47:58.277788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277745    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache" (OuterVolumeSpecName: "model-cache") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:58.277852 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home" (OuterVolumeSpecName: "home") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:58.277928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277902    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:58.277981 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277930    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.277981 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.277944    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.279550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.279523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm" (OuterVolumeSpecName: "dshm") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:58.279550 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.279535    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4" (OuterVolumeSpecName: "kube-api-access-7fcz4") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "kube-api-access-7fcz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:58.279713 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.279628    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8bb25803-320e-4788-9967-c5970c84de80" (UID: "8bb25803-320e-4788-9967-c5970c84de80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:47:58.323826 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.323796    2564 generic.go:356] "Generic (PLEG): container finished" podID="8bb25803-320e-4788-9967-c5970c84de80" containerID="509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c" exitCode=0
Jul 27 17:47:58.323950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.323839    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"
Jul 27 17:47:58.323950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.323859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" event={"ID":"8bb25803-320e-4788-9967-c5970c84de80","Type":"ContainerDied","Data":"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"}
Jul 27 17:47:58.323950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.323922    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5" event={"ID":"8bb25803-320e-4788-9967-c5970c84de80","Type":"ContainerDied","Data":"861a6cfb87f2cc984274cc054fd0ca702e8b70a586a345b04ab72ed81a6b729a"}
Jul 27 17:47:58.323950 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.323947    2564 scope.go:117] "RemoveContainer" containerID="509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"
Jul 27 17:47:58.332769 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.332753    2564 scope.go:117] "RemoveContainer" containerID="509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"
Jul 27 17:47:58.333047 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:47:58.333029    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c\": container with ID starting with 509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c not found: ID does not exist" containerID="509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"
Jul 27 17:47:58.333125 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.333060    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c"} err="failed to get container status \"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c\": rpc error: code = NotFound desc = could not find container \"509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c\": container with ID starting with 509152cbb34f35aa1ea8431222f9e2f593d64d5fb178d24896aca63c1e98c30c not found: ID does not exist"
Jul 27 17:47:58.359337 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.359315    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:47:58.363091 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.363070    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-004618/rate-test-model-004618-kserve-79bbf4d5cf-s4ww5"]
Jul 27 17:47:58.379000 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.378978    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcz4\" (UniqueName: \"kubernetes.io/projected/8bb25803-320e-4788-9967-c5970c84de80-kube-api-access-7fcz4\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.379000 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.378998    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.379106 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.379009    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb25803-320e-4788-9967-c5970c84de80-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.379106 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.379017    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb25803-320e-4788-9967-c5970c84de80-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:58.658778 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.658746    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb25803-320e-4788-9967-c5970c84de80" path="/var/lib/kubelet/pods/8bb25803-320e-4788-9967-c5970c84de80/volumes"
Jul 27 17:47:58.742164 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.742130    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerName="main" containerID="cri-o://dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f" gracePeriod=45
Jul 27 17:47:58.998003 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:58.997978    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:47:59.084273 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084242    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084288    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbbdf\" (UniqueName: \"kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084323    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084374    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084454 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084424    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home\") pod \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\" (UID: \"dacc3ddf-fedb-49f8-a323-992d4f0c82f4\") "
Jul 27 17:47:59.084697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084636    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:59.084697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084646    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:59.084697 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.084687    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home" (OuterVolumeSpecName: "home") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:59.086468 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.086440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm" (OuterVolumeSpecName: "dshm") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:59.086468 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.086464    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:47:59.086608 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.086480    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf" (OuterVolumeSpecName: "kube-api-access-wbbdf") pod "dacc3ddf-fedb-49f8-a323-992d4f0c82f4" (UID: "dacc3ddf-fedb-49f8-a323-992d4f0c82f4"). InnerVolumeSpecName "kube-api-access-wbbdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185100    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185121    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185130    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185138    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbdf\" (UniqueName: \"kubernetes.io/projected/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-kube-api-access-wbbdf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185148    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.185159 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.185156    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacc3ddf-fedb-49f8-a323-992d4f0c82f4-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:47:59.330398 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.330372    2564 generic.go:356] "Generic (PLEG): container finished" podID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerID="dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f" exitCode=0
Jul 27 17:47:59.330524 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.330430    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"
Jul 27 17:47:59.330524 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.330433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" event={"ID":"dacc3ddf-fedb-49f8-a323-992d4f0c82f4","Type":"ContainerDied","Data":"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"}
Jul 27 17:47:59.330601 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.330530    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" event={"ID":"dacc3ddf-fedb-49f8-a323-992d4f0c82f4","Type":"ContainerDied","Data":"29c0e013f0ee288e4471fd80b857904775654c27efbd95701ecf52f03d622e60"}
Jul 27 17:47:59.330601 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.330547    2564 scope.go:117] "RemoveContainer" containerID="dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"
Jul 27 17:47:59.339287 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.339263    2564 scope.go:117] "RemoveContainer" containerID="dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"
Jul 27 17:47:59.339549 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:47:59.339531    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f\": container with ID starting with dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f not found: ID does not exist" containerID="dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"
Jul 27 17:47:59.339612 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.339557    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f"} err="failed to get container status \"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f\": rpc error: code = NotFound desc = could not find container \"dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f\": container with ID starting with dfb51d82d1a28f1e3997a529d3d9365d0706d6f20e6fbbdadeb4afbac94aad8f not found: ID does not exist"
Jul 27 17:47:59.361956 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.361936    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:47:59.368768 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:47:59.368749    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn"]
Jul 27 17:48:00.658664 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:00.658632    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" path="/var/lib/kubelet/pods/dacc3ddf-fedb-49f8-a323-992d4f0c82f4/volumes"
Jul 27 17:48:03.956839 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:03.956782    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-509a5d/rate-test-model-509a5d-kserve-787d56c755-gq4jn" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:48:07.127034 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.126999    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:48:07.127576 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127547    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerName="main"
Jul 27 17:48:07.127576 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127574    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerName="main"
Jul 27 17:48:07.127736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127607    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb25803-320e-4788-9967-c5970c84de80" containerName="main"
Jul 27 17:48:07.127736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127616    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb25803-320e-4788-9967-c5970c84de80" containerName="main"
Jul 27 17:48:07.127736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127709    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb25803-320e-4788-9967-c5970c84de80" containerName="main"
Jul 27 17:48:07.127736 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.127718    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacc3ddf-fedb-49f8-a323-992d4f0c82f4" containerName="main"
Jul 27 17:48:07.130780 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.130760    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.140654 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.140635    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d401e5-serving-cert\""
Jul 27 17:48:07.216001 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.215978    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:48:07.253095 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.253071    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.253194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.253110    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.253194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.253155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tqb\" (UniqueName: \"kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.354090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.354067    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.354196 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.354098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.354196 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.354132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tqb\" (UniqueName: \"kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.354709 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.354686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.356508 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.356488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.370284 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.370255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tqb\" (UniqueName: \"kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb\") pod \"maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.440971 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.440951    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:07.584471 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:48:07.584439    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1c26c8_8d6f_4369_9644_c2389794d24e.slice/crio-546e8993e97c26e6e91a2e81eb50fef9a05d1df3668ae086f3a1b629e89fa6c7 WatchSource:0}: Error finding container 546e8993e97c26e6e91a2e81eb50fef9a05d1df3668ae086f3a1b629e89fa6c7: Status 404 returned error can't find the container with id 546e8993e97c26e6e91a2e81eb50fef9a05d1df3668ae086f3a1b629e89fa6c7
Jul 27 17:48:07.584606 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.584469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:48:07.585791 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:07.585775    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:48:08.286612 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.286577    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:48:08.290046 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.290030    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.294464 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.294445    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d401e5-dockercfg-pqlv8\""
Jul 27 17:48:08.295194 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.295177    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d401e5\""
Jul 27 17:48:08.306061 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.306042    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:48:08.375440 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.375407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" event={"ID":"fe1c26c8-8d6f-4369-9644-c2389794d24e","Type":"ContainerStarted","Data":"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"}
Jul 27 17:48:08.375573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.375459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" event={"ID":"fe1c26c8-8d6f-4369-9644-c2389794d24e","Type":"ContainerStarted","Data":"546e8993e97c26e6e91a2e81eb50fef9a05d1df3668ae086f3a1b629e89fa6c7"}
Jul 27 17:48:08.375573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.375482    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:08.444291 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.444252    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" podStartSLOduration=0.893024297 podStartE2EDuration="1.444236963s" totalImagesPullingTime="551.212666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:08.443121041 +0000 UTC m=+3108.275486029" watchObservedRunningTime="2026-07-27 17:48:08.444236963 +0000 UTC m=+3108.276601982"
Jul 27 17:48:08.464455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.464426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.464563 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.464460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87m4q\" (UniqueName: \"kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.565856 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.565826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m4q\" (UniqueName: \"kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.566017 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.565970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.566618 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.566600    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.583573 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.583546    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m4q\" (UniqueName: \"kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q\") pod \"payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.600311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.600278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:08.740421 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:08.740397    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:48:08.742629 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:48:08.742602    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15de0f11_7157_430c_889b_5649c416ab1a.slice/crio-2f452f0d467b5319d86fdfc6acea0a7dc2f41640c7f6bd3d6d19239add4077ed WatchSource:0}: Error finding container 2f452f0d467b5319d86fdfc6acea0a7dc2f41640c7f6bd3d6d19239add4077ed: Status 404 returned error can't find the container with id 2f452f0d467b5319d86fdfc6acea0a7dc2f41640c7f6bd3d6d19239add4077ed
Jul 27 17:48:09.381479 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:09.381447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" event={"ID":"15de0f11-7157-430c-889b-5649c416ab1a","Type":"ContainerStarted","Data":"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"}
Jul 27 17:48:09.381856 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:09.381500    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:09.381856 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:09.381518    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" event={"ID":"15de0f11-7157-430c-889b-5649c416ab1a","Type":"ContainerStarted","Data":"2f452f0d467b5319d86fdfc6acea0a7dc2f41640c7f6bd3d6d19239add4077ed"}
Jul 27 17:48:09.403138 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:09.403100    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" podStartSLOduration=1.403084916 podStartE2EDuration="1.403084916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:09.402437096 +0000 UTC m=+3109.234802072" watchObservedRunningTime="2026-07-27 17:48:09.403084916 +0000 UTC m=+3109.235449891"
Jul 27 17:48:14.387373 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:14.387342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:48:15.792439 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:15.792409    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:48:20.388065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:20.388031    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:48:25.977991 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:25.977958    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:48:25.981357 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:25.981336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:25.984794 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:25.984776    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3bb3ae-serving-cert\""
Jul 27 17:48:25.997602 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:25.997580    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:48:26.013347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.013323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.013437 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.013360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.013504 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.013478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4g6\" (UniqueName: \"kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.114364 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.114334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.114539 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.114413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4g6\" (UniqueName: \"kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.114539 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.114451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.115118 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.115095    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.116820 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.116794    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.136788 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.136758    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4g6\" (UniqueName: \"kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6\") pod \"maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.291790 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.291733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:26.447076 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.447050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:48:26.448633 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:48:26.448608    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31045e1_5a1e_4808_85be_901cc449259c.slice/crio-4a7693d1044d77d3cea4d7e8339e7c0b8277d7c1a1830dab564b0d952435601b WatchSource:0}: Error finding container 4a7693d1044d77d3cea4d7e8339e7c0b8277d7c1a1830dab564b0d952435601b: Status 404 returned error can't find the container with id 4a7693d1044d77d3cea4d7e8339e7c0b8277d7c1a1830dab564b0d952435601b
Jul 27 17:48:26.463317 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:26.463292    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" event={"ID":"a31045e1-5a1e-4808-85be-901cc449259c","Type":"ContainerStarted","Data":"4a7693d1044d77d3cea4d7e8339e7c0b8277d7c1a1830dab564b0d952435601b"}
Jul 27 17:48:27.154874 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.154404    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:48:27.158430 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.158412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.164064 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.164017    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3bb3ae-dockercfg-gfz84\""
Jul 27 17:48:27.164198 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.164109    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3bb3ae\""
Jul 27 17:48:27.181651 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.181629    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:48:27.225157 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.225133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.225258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.225164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2l5x\" (UniqueName: \"kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.326675 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.326641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.326832 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.326691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2l5x\" (UniqueName: \"kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.327350 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.327331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.336169 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.336149    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2l5x\" (UniqueName: \"kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x\") pod \"payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.468240 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.468168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:27.469796 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.469770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" event={"ID":"a31045e1-5a1e-4808-85be-901cc449259c","Type":"ContainerStarted","Data":"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"}
Jul 27 17:48:27.469945 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.469914    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:27.497049 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.497003    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" podStartSLOduration=2.165688674 podStartE2EDuration="2.496984333s" totalImagesPullingTime="331.295659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:27.495037822 +0000 UTC m=+3127.327402798" watchObservedRunningTime="2026-07-27 17:48:27.496984333 +0000 UTC m=+3127.329349308"
Jul 27 17:48:27.602245 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:27.602214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:48:27.604210 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:48:27.604186    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ba8dcc_ca02_4ac2_a589_3c6e2276cb0a.slice/crio-dce69807a980f47d0c64520e4986e9b5887e82d3b9e2fb9066bc674b6ab5b9f9 WatchSource:0}: Error finding container dce69807a980f47d0c64520e4986e9b5887e82d3b9e2fb9066bc674b6ab5b9f9: Status 404 returned error can't find the container with id dce69807a980f47d0c64520e4986e9b5887e82d3b9e2fb9066bc674b6ab5b9f9
Jul 27 17:48:28.478805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:28.478769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" event={"ID":"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a","Type":"ContainerStarted","Data":"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"}
Jul 27 17:48:28.479221 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:28.478827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" event={"ID":"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a","Type":"ContainerStarted","Data":"dce69807a980f47d0c64520e4986e9b5887e82d3b9e2fb9066bc674b6ab5b9f9"}
Jul 27 17:48:28.503747 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:28.503700    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" podStartSLOduration=1.5036846750000001 podStartE2EDuration="1.503684675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:28.502600588 +0000 UTC m=+3128.334965565" watchObservedRunningTime="2026-07-27 17:48:28.503684675 +0000 UTC m=+3128.336049651"
Jul 27 17:48:29.483217 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:29.483192    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:33.484226 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:33.484196    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:48:34.886929 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:34.886902    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:48:40.489917 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:40.489861    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:48:54.822413 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.822340    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:48:54.826239 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.826222    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.830893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.830852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d401e5\"/\"default-dockercfg-p6kdx\""
Jul 27 17:48:54.830893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.830850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d401e5\"/\"ipp-route-d401e5-kserve-self-signed-certs\""
Jul 27 17:48:54.831112 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.830906    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d401e5\"/\"kube-root-ca.crt\""
Jul 27 17:48:54.831112 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.830851    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d401e5\"/\"openshift-service-ca.crt\""
Jul 27 17:48:54.836692 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.836672    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:48:54.945154 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.945300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.945300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945180    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv65h\" (UniqueName: \"kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.945300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945287    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.945455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:54.945455 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:54.945352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046153 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046115    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046208    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv65h\" (UniqueName: \"kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046347 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046543 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046357    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046703 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046675    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046779 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.046779 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.046752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.048586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.048559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.048682 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.048595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.056707 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.056686    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv65h\" (UniqueName: \"kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h\") pod \"ipp-route-d401e5-kserve-76cbd89cdc-dx4gw\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") " pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.136475 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.136383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:48:55.272257 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.272231    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:48:55.273754 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:48:55.273727    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ae4209_2bd3_40d8_9a1c_5aafe2ecb396.slice/crio-369cbae916a8df770cf3fdea1769c222cb14c18da58bfece2644da5c79704535 WatchSource:0}: Error finding container 369cbae916a8df770cf3fdea1769c222cb14c18da58bfece2644da5c79704535: Status 404 returned error can't find the container with id 369cbae916a8df770cf3fdea1769c222cb14c18da58bfece2644da5c79704535
Jul 27 17:48:55.606403 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.606371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" event={"ID":"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396","Type":"ContainerStarted","Data":"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"}
Jul 27 17:48:55.606403 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:55.606410    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" event={"ID":"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396","Type":"ContainerStarted","Data":"369cbae916a8df770cf3fdea1769c222cb14c18da58bfece2644da5c79704535"}
Jul 27 17:48:58.307722 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:58.307670    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" podStartSLOduration=4.307651575 podStartE2EDuration="4.307651575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:55.629770785 +0000 UTC m=+3155.462135762" watchObservedRunningTime="2026-07-27 17:48:58.307651575 +0000 UTC m=+3158.140016561"
Jul 27 17:48:58.309547 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:48:58.309521    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:49:05.136779 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:05.136731    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:49:05.137230 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:05.136800    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:49:05.153864 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:05.153837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:49:05.668980 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:05.668947    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:49:27.691463 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:27.691428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:49:39.590500 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:39.590470    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s_15de0f11-7157-430c-889b-5649c416ab1a/payload-processing/0.log"
Jul 27 17:49:44.473253 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.473218    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:49:44.476787 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.476767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.480019 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.479997    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4df9d8\""
Jul 27 17:49:44.480142 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.480066    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4df9d8-dockercfg-jpqpt\""
Jul 27 17:49:44.489595 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.489571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:49:44.558273 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.558239    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pbz\" (UniqueName: \"kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.558448 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.558336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.659139 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.659106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pbz\" (UniqueName: \"kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.659295 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.659167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.659777 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.659757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.668895 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.668857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pbz\" (UniqueName: \"kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz\") pod \"payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.787383 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.787304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:44.924567 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:44.924537    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:49:44.926951 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:49:44.926918    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86808ed5_0fe5_41d1_9f6f_d8cbbea0b5a4.slice/crio-b49e8fa77012e30e0de114e26834d36ab2c4293ccc96cead256a0cdf1600699c WatchSource:0}: Error finding container b49e8fa77012e30e0de114e26834d36ab2c4293ccc96cead256a0cdf1600699c: Status 404 returned error can't find the container with id b49e8fa77012e30e0de114e26834d36ab2c4293ccc96cead256a0cdf1600699c
Jul 27 17:49:45.848661 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:45.848628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" event={"ID":"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4","Type":"ContainerStarted","Data":"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"}
Jul 27 17:49:45.849038 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:45.848690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:49:45.849038 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:45.848709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" event={"ID":"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4","Type":"ContainerStarted","Data":"b49e8fa77012e30e0de114e26834d36ab2c4293ccc96cead256a0cdf1600699c"}
Jul 27 17:49:45.870579 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:45.870530    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" podStartSLOduration=1.870514486 podStartE2EDuration="1.870514486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:49:45.869681893 +0000 UTC m=+3205.702046886" watchObservedRunningTime="2026-07-27 17:49:45.870514486 +0000 UTC m=+3205.702879462"
Jul 27 17:49:52.398117 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:52.398085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:49:56.856044 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:49:56.856015    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:50:09.699730 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.699694    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:50:09.700276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.699998    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" podUID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" containerName="payload-processing" containerID="cri-o://c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5" gracePeriod=30
Jul 27 17:50:09.955352 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.955296    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:50:09.969390 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.969363    2564 generic.go:356] "Generic (PLEG): container finished" podID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" containerID="c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5" exitCode=0
Jul 27 17:50:09.969502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.969400    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"
Jul 27 17:50:09.969502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.969451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" event={"ID":"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4","Type":"ContainerDied","Data":"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"}
Jul 27 17:50:09.969502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.969499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5" event={"ID":"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4","Type":"ContainerDied","Data":"b49e8fa77012e30e0de114e26834d36ab2c4293ccc96cead256a0cdf1600699c"}
Jul 27 17:50:09.969610 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.969523    2564 scope.go:117] "RemoveContainer" containerID="c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"
Jul 27 17:50:09.978510 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.978493    2564 scope.go:117] "RemoveContainer" containerID="c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"
Jul 27 17:50:09.978831 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:50:09.978800    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5\": container with ID starting with c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5 not found: ID does not exist" containerID="c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"
Jul 27 17:50:09.978949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:09.978834    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5"} err="failed to get container status \"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5\": rpc error: code = NotFound desc = could not find container \"c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5\": container with ID starting with c89c08406d02ea0e855f40a85a599a9ef4e4570bc21226a9833f9965c39d61f5 not found: ID does not exist"
Jul 27 17:50:10.067610 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.067582    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume\") pod \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") "
Jul 27 17:50:10.067824 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.067655    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4pbz\" (UniqueName: \"kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz\") pod \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\" (UID: \"86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4\") "
Jul 27 17:50:10.067952 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.067932    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" (UID: "86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:50:10.069616 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.069586    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz" (OuterVolumeSpecName: "kube-api-access-j4pbz") pod "86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" (UID: "86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4"). InnerVolumeSpecName "kube-api-access-j4pbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:50:10.168894 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.168853    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:10.168894 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.168873    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pbz\" (UniqueName: \"kubernetes.io/projected/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4-kube-api-access-j4pbz\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:10.292949 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.292920    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:50:10.295242 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.295217    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-4df9d8-789874bc8-4cxk5"]
Jul 27 17:50:10.658781 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:10.658746    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" path="/var/lib/kubelet/pods/86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4/volumes"
Jul 27 17:50:14.188975 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:14.188871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:50:39.295345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:39.295308    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:50:40.202922 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.202892    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:50:40.203108 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.203082    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" podUID="fe1c26c8-8d6f-4369-9644-c2389794d24e" containerName="maas-api" containerID="cri-o://1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0" gracePeriod=30
Jul 27 17:50:40.309325 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.309293    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:50:40.309767 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.309718    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" podUID="15de0f11-7157-430c-889b-5649c416ab1a" containerName="payload-processing" containerID="cri-o://e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35" gracePeriod=30
Jul 27 17:50:40.387349 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.387310    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" podUID="15de0f11-7157-430c-889b-5649c416ab1a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.73:9004: connect: connection refused"
Jul 27 17:50:40.461477 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.461421    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:50:40.517674 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.517640    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca\") pod \"fe1c26c8-8d6f-4369-9644-c2389794d24e\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") "
Jul 27 17:50:40.517674 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.517677    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7tqb\" (UniqueName: \"kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb\") pod \"fe1c26c8-8d6f-4369-9644-c2389794d24e\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") "
Jul 27 17:50:40.517918 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.517716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls\") pod \"fe1c26c8-8d6f-4369-9644-c2389794d24e\" (UID: \"fe1c26c8-8d6f-4369-9644-c2389794d24e\") "
Jul 27 17:50:40.518018 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.517994    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe1c26c8-8d6f-4369-9644-c2389794d24e" (UID: "fe1c26c8-8d6f-4369-9644-c2389794d24e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:50:40.519681 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.519656    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb" (OuterVolumeSpecName: "kube-api-access-r7tqb") pod "fe1c26c8-8d6f-4369-9644-c2389794d24e" (UID: "fe1c26c8-8d6f-4369-9644-c2389794d24e"). InnerVolumeSpecName "kube-api-access-r7tqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:50:40.519681 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.519677    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe1c26c8-8d6f-4369-9644-c2389794d24e" (UID: "fe1c26c8-8d6f-4369-9644-c2389794d24e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:50:40.576774 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.576754    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:50:40.618845 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.618819    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87m4q\" (UniqueName: \"kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q\") pod \"15de0f11-7157-430c-889b-5649c416ab1a\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") "
Jul 27 17:50:40.619019 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.618851    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume\") pod \"15de0f11-7157-430c-889b-5649c416ab1a\" (UID: \"15de0f11-7157-430c-889b-5649c416ab1a\") "
Jul 27 17:50:40.619121 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.619107    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe1c26c8-8d6f-4369-9644-c2389794d24e-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:40.619184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.619125    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1c26c8-8d6f-4369-9644-c2389794d24e-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:40.619184 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.619137    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tqb\" (UniqueName: \"kubernetes.io/projected/fe1c26c8-8d6f-4369-9644-c2389794d24e-kube-api-access-r7tqb\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:40.619354 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.619327    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15de0f11-7157-430c-889b-5649c416ab1a" (UID: "15de0f11-7157-430c-889b-5649c416ab1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:50:40.620922 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.620899    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q" (OuterVolumeSpecName: "kube-api-access-87m4q") pod "15de0f11-7157-430c-889b-5649c416ab1a" (UID: "15de0f11-7157-430c-889b-5649c416ab1a"). InnerVolumeSpecName "kube-api-access-87m4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:50:40.719869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.719800    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87m4q\" (UniqueName: \"kubernetes.io/projected/15de0f11-7157-430c-889b-5649c416ab1a-kube-api-access-87m4q\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:40.719869 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:40.719824    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15de0f11-7157-430c-889b-5649c416ab1a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:50:41.115203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.115128    2564 generic.go:356] "Generic (PLEG): container finished" podID="15de0f11-7157-430c-889b-5649c416ab1a" containerID="e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35" exitCode=0
Jul 27 17:50:41.115203 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.115172    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"
Jul 27 17:50:41.115492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.115211    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" event={"ID":"15de0f11-7157-430c-889b-5649c416ab1a","Type":"ContainerDied","Data":"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"}
Jul 27 17:50:41.115492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.115265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s" event={"ID":"15de0f11-7157-430c-889b-5649c416ab1a","Type":"ContainerDied","Data":"2f452f0d467b5319d86fdfc6acea0a7dc2f41640c7f6bd3d6d19239add4077ed"}
Jul 27 17:50:41.115492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.115290    2564 scope.go:117] "RemoveContainer" containerID="e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"
Jul 27 17:50:41.116549 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.116524    2564 generic.go:356] "Generic (PLEG): container finished" podID="fe1c26c8-8d6f-4369-9644-c2389794d24e" containerID="1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0" exitCode=0
Jul 27 17:50:41.116659 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.116562    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"
Jul 27 17:50:41.116659 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.116602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" event={"ID":"fe1c26c8-8d6f-4369-9644-c2389794d24e","Type":"ContainerDied","Data":"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"}
Jul 27 17:50:41.116659 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.116639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x" event={"ID":"fe1c26c8-8d6f-4369-9644-c2389794d24e","Type":"ContainerDied","Data":"546e8993e97c26e6e91a2e81eb50fef9a05d1df3668ae086f3a1b629e89fa6c7"}
Jul 27 17:50:41.123830 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.123807    2564 scope.go:117] "RemoveContainer" containerID="e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"
Jul 27 17:50:41.124188 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:50:41.124166    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35\": container with ID starting with e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35 not found: ID does not exist" containerID="e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"
Jul 27 17:50:41.124281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.124194    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35"} err="failed to get container status \"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35\": rpc error: code = NotFound desc = could not find container \"e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35\": container with ID starting with e3fb91e3065de40c225e9ff3e0640babc2906161b3e17b3ced16be8718cc2d35 not found: ID does not exist"
Jul 27 17:50:41.124281 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.124210    2564 scope.go:117] "RemoveContainer" containerID="1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"
Jul 27 17:50:41.131669 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.131651    2564 scope.go:117] "RemoveContainer" containerID="1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"
Jul 27 17:50:41.131923 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:50:41.131901    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0\": container with ID starting with 1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0 not found: ID does not exist" containerID="1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"
Jul 27 17:50:41.131998 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.131932    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0"} err="failed to get container status \"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0\": rpc error: code = NotFound desc = could not find container \"1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0\": container with ID starting with 1224eb514e23684f361cd7d339d436a1b8e6bb089d611e861ddda5328b2d8fd0 not found: ID does not exist"
Jul 27 17:50:41.136492 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.136472    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:50:41.142511 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.142493    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d401e5-58fb67bd7f-nx85x"]
Jul 27 17:50:41.159471 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.159452    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:50:41.164027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:41.164007    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d401e5-55c47b5d9b-zxg5s"]
Jul 27 17:50:42.658673 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:42.658642    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15de0f11-7157-430c-889b-5649c416ab1a" path="/var/lib/kubelet/pods/15de0f11-7157-430c-889b-5649c416ab1a/volumes"
Jul 27 17:50:42.659058 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:42.659016    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1c26c8-8d6f-4369-9644-c2389794d24e" path="/var/lib/kubelet/pods/fe1c26c8-8d6f-4369-9644-c2389794d24e/volumes"
Jul 27 17:50:49.722419 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:49.722387    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:50:50.128686 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:50:50.128588    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:51:05.147966 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.147896    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" podUID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" containerName="main" containerID="cri-o://2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5" gracePeriod=45
Jul 27 17:51:05.400635 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.400586    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:51:05.514893 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.514857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.514927    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.514949    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515027 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.514999    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv65h\" (UniqueName: \"kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515141 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515039    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515141 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515064    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs\") pod \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\" (UID: \"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396\") "
Jul 27 17:51:05.515229 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home" (OuterVolumeSpecName: "home") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:05.515229 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515165    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache" (OuterVolumeSpecName: "model-cache") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:05.515229 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515209    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:05.515360 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515344    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tmp-dir\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:05.515424 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515365    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-home\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:05.515424 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.515378    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-model-cache\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:05.517205 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.517182    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm" (OuterVolumeSpecName: "dshm") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:05.517205 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.517196    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:05.517309 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.517186    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h" (OuterVolumeSpecName: "kube-api-access-zv65h") pod "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" (UID: "72ae4209-2bd3-40d8-9a1c-5aafe2ecb396"). InnerVolumeSpecName "kube-api-access-zv65h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:05.616501 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.616474    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv65h\" (UniqueName: \"kubernetes.io/projected/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-kube-api-access-zv65h\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:05.616501 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.616498    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-dshm\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:05.616679 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:05.616512    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396-tls-certs\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:06.242773 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.242744    2564 generic.go:356] "Generic (PLEG): container finished" podID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" containerID="2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5" exitCode=0
Jul 27 17:51:06.243193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.242781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" event={"ID":"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396","Type":"ContainerDied","Data":"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"}
Jul 27 17:51:06.243193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.242789    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"
Jul 27 17:51:06.243193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.242812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw" event={"ID":"72ae4209-2bd3-40d8-9a1c-5aafe2ecb396","Type":"ContainerDied","Data":"369cbae916a8df770cf3fdea1769c222cb14c18da58bfece2644da5c79704535"}
Jul 27 17:51:06.243193 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.242829    2564 scope.go:117] "RemoveContainer" containerID="2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"
Jul 27 17:51:06.256924 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.256906    2564 scope.go:117] "RemoveContainer" containerID="2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"
Jul 27 17:51:06.257156 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:51:06.257137    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5\": container with ID starting with 2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5 not found: ID does not exist" containerID="2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"
Jul 27 17:51:06.257206 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.257164    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5"} err="failed to get container status \"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5\": rpc error: code = NotFound desc = could not find container \"2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5\": container with ID starting with 2936eb6e9a4a5a18ecaf6e4e94d6a034ae7a85d4858bafc98c2daee93ad615d5 not found: ID does not exist"
Jul 27 17:51:06.270354 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.270329    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:51:06.275182 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.275159    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d401e5/ipp-route-d401e5-kserve-76cbd89cdc-dx4gw"]
Jul 27 17:51:06.659042 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:06.659014    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" path="/var/lib/kubelet/pods/72ae4209-2bd3-40d8-9a1c-5aafe2ecb396/volumes"
Jul 27 17:51:20.839785 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:20.839762    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:51:20.847025 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:20.847001    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:51:50.882144 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:50.882068    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:51:50.882544 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:50.882275    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" podUID="a31045e1-5a1e-4808-85be-901cc449259c" containerName="maas-api" containerID="cri-o://9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb" gracePeriod=30
Jul 27 17:51:50.956338 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:50.956311    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:51:50.956537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:50.956514    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" podUID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" containerName="payload-pre-processing" containerID="cri-o://8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d" gracePeriod=30
Jul 27 17:51:51.201776 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.201752    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:51:51.230086 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.230062    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:51:51.276951 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.276928    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb4g6\" (UniqueName: \"kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6\") pod \"a31045e1-5a1e-4808-85be-901cc449259c\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") "
Jul 27 17:51:51.277065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.276994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca\") pod \"a31045e1-5a1e-4808-85be-901cc449259c\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") "
Jul 27 17:51:51.277065 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.277053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2l5x\" (UniqueName: \"kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x\") pod \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") "
Jul 27 17:51:51.277138 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.277088    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume\") pod \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\" (UID: \"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a\") "
Jul 27 17:51:51.277138 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.277114    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls\") pod \"a31045e1-5a1e-4808-85be-901cc449259c\" (UID: \"a31045e1-5a1e-4808-85be-901cc449259c\") "
Jul 27 17:51:51.277472 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.277441    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a31045e1-5a1e-4808-85be-901cc449259c" (UID: "a31045e1-5a1e-4808-85be-901cc449259c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:51:51.277578 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.277515    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" (UID: "e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:51:51.279087 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.279054    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a31045e1-5a1e-4808-85be-901cc449259c" (UID: "a31045e1-5a1e-4808-85be-901cc449259c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:51.279180 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.279125    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6" (OuterVolumeSpecName: "kube-api-access-xb4g6") pod "a31045e1-5a1e-4808-85be-901cc449259c" (UID: "a31045e1-5a1e-4808-85be-901cc449259c"). InnerVolumeSpecName "kube-api-access-xb4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:51.279180 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.279143    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x" (OuterVolumeSpecName: "kube-api-access-m2l5x") pod "e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" (UID: "e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a"). InnerVolumeSpecName "kube-api-access-m2l5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:51.377701 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.377670    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31045e1-5a1e-4808-85be-901cc449259c-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:51.377701 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.377696    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2l5x\" (UniqueName: \"kubernetes.io/projected/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-kube-api-access-m2l5x\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:51.377843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.377709    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:51.377843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.377723    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a31045e1-5a1e-4808-85be-901cc449259c-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:51.377843 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.377735    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4g6\" (UniqueName: \"kubernetes.io/projected/a31045e1-5a1e-4808-85be-901cc449259c-kube-api-access-xb4g6\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:51:51.455430 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.455366    2564 generic.go:356] "Generic (PLEG): container finished" podID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" containerID="8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d" exitCode=0
Jul 27 17:51:51.455430 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.455411    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"
Jul 27 17:51:51.455430 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.455413    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" event={"ID":"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a","Type":"ContainerDied","Data":"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"}
Jul 27 17:51:51.455631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.455480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28" event={"ID":"e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a","Type":"ContainerDied","Data":"dce69807a980f47d0c64520e4986e9b5887e82d3b9e2fb9066bc674b6ab5b9f9"}
Jul 27 17:51:51.455631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.455512    2564 scope.go:117] "RemoveContainer" containerID="8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"
Jul 27 17:51:51.456805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.456782    2564 generic.go:356] "Generic (PLEG): container finished" podID="a31045e1-5a1e-4808-85be-901cc449259c" containerID="9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb" exitCode=0
Jul 27 17:51:51.456927 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.456818    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"
Jul 27 17:51:51.456927 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.456859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" event={"ID":"a31045e1-5a1e-4808-85be-901cc449259c","Type":"ContainerDied","Data":"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"}
Jul 27 17:51:51.456927 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.456919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf" event={"ID":"a31045e1-5a1e-4808-85be-901cc449259c","Type":"ContainerDied","Data":"4a7693d1044d77d3cea4d7e8339e7c0b8277d7c1a1830dab564b0d952435601b"}
Jul 27 17:51:51.466051 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.466032    2564 scope.go:117] "RemoveContainer" containerID="8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"
Jul 27 17:51:51.466288 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:51:51.466269    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d\": container with ID starting with 8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d not found: ID does not exist" containerID="8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"
Jul 27 17:51:51.466367 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.466301    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d"} err="failed to get container status \"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d\": rpc error: code = NotFound desc = could not find container \"8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d\": container with ID starting with 8d2218a33675f23ae4cdab2cc18ee0ce5ffa0bc2d9b49f62a46630c83a2dfd1d not found: ID does not exist"
Jul 27 17:51:51.466367 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.466324    2564 scope.go:117] "RemoveContainer" containerID="9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"
Jul 27 17:51:51.473586 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.473568    2564 scope.go:117] "RemoveContainer" containerID="9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"
Jul 27 17:51:51.473828 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:51:51.473812    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb\": container with ID starting with 9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb not found: ID does not exist" containerID="9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"
Jul 27 17:51:51.473870 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.473835    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb"} err="failed to get container status \"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb\": rpc error: code = NotFound desc = could not find container \"9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb\": container with ID starting with 9a1af0991a1b3addd9cd44e5d6d8dfc50636656a00b1c8dd7da913997fa849cb not found: ID does not exist"
Jul 27 17:51:51.481180 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.481159    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:51:51.485731 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.485712    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3bb3ae-66f5b4fdc6-8hd28"]
Jul 27 17:51:51.504216 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.504193    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:51:51.508336 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:51.508314    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3bb3ae-6896f886cc-gxjqf"]
Jul 27 17:51:52.657631 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:52.657597    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31045e1-5a1e-4808-85be-901cc449259c" path="/var/lib/kubelet/pods/a31045e1-5a1e-4808-85be-901cc449259c/volumes"
Jul 27 17:51:52.658089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:52.657947    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" path="/var/lib/kubelet/pods/e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a/volumes"
Jul 27 17:51:54.085930 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:51:54.085899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:52:13.594394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.594359    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:52:13.594903 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.594573    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" podUID="ea4bd945-4e03-4f68-a850-3d45b597001d" containerName="maas-api" containerID="cri-o://e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f" gracePeriod=30
Jul 27 17:52:13.661060 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.661028    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:52:13.661343 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.661293    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" podUID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" containerName="payload-processing" containerID="cri-o://74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7" gracePeriod=30
Jul 27 17:52:13.672543 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.672516    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:52:13.672743 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.672720    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" podUID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" containerName="payload-pre-processing" containerID="cri-o://2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858" gracePeriod=30
Jul 27 17:52:13.929345 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.929318    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:52:13.974551 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.974521    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:52:13.979167 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:13.979150    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:52:14.054658 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054634    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2q22\" (UniqueName: \"kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22\") pod \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") "
Jul 27 17:52:14.054805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054687    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5fkd\" (UniqueName: \"kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd\") pod \"ea4bd945-4e03-4f68-a850-3d45b597001d\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") "
Jul 27 17:52:14.054805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054708    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume\") pod \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\" (UID: \"3f7dfe4c-4ecb-444f-950e-53ed7b72917b\") "
Jul 27 17:52:14.054805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054731    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwpmf\" (UniqueName: \"kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf\") pod \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") "
Jul 27 17:52:14.054805 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054796    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume\") pod \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\" (UID: \"4f6e78f4-f2df-43b5-874a-0095dd4a1f17\") "
Jul 27 17:52:14.055072 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054821    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca\") pod \"ea4bd945-4e03-4f68-a850-3d45b597001d\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") "
Jul 27 17:52:14.055072 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.054858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls\") pod \"ea4bd945-4e03-4f68-a850-3d45b597001d\" (UID: \"ea4bd945-4e03-4f68-a850-3d45b597001d\") "
Jul 27 17:52:14.055301 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.055219    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea4bd945-4e03-4f68-a850-3d45b597001d" (UID: "ea4bd945-4e03-4f68-a850-3d45b597001d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:52:14.055301 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.055226    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f6e78f4-f2df-43b5-874a-0095dd4a1f17" (UID: "4f6e78f4-f2df-43b5-874a-0095dd4a1f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:52:14.055438 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.055344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f7dfe4c-4ecb-444f-950e-53ed7b72917b" (UID: "3f7dfe4c-4ecb-444f-950e-53ed7b72917b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:52:14.056931 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.056895    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf" (OuterVolumeSpecName: "kube-api-access-jwpmf") pod "4f6e78f4-f2df-43b5-874a-0095dd4a1f17" (UID: "4f6e78f4-f2df-43b5-874a-0095dd4a1f17"). InnerVolumeSpecName "kube-api-access-jwpmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:14.057188 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.057169    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd" (OuterVolumeSpecName: "kube-api-access-j5fkd") pod "ea4bd945-4e03-4f68-a850-3d45b597001d" (UID: "ea4bd945-4e03-4f68-a850-3d45b597001d"). InnerVolumeSpecName "kube-api-access-j5fkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:14.057300 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.057280    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea4bd945-4e03-4f68-a850-3d45b597001d" (UID: "ea4bd945-4e03-4f68-a850-3d45b597001d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:52:14.057344 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.057281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22" (OuterVolumeSpecName: "kube-api-access-s2q22") pod "3f7dfe4c-4ecb-444f-950e-53ed7b72917b" (UID: "3f7dfe4c-4ecb-444f-950e-53ed7b72917b"). InnerVolumeSpecName "kube-api-access-s2q22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155928    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155949    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4bd945-4e03-4f68-a850-3d45b597001d-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155958    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4bd945-4e03-4f68-a850-3d45b597001d-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155967    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2q22\" (UniqueName: \"kubernetes.io/projected/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-kube-api-access-s2q22\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155975    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fkd\" (UniqueName: \"kubernetes.io/projected/ea4bd945-4e03-4f68-a850-3d45b597001d-kube-api-access-j5fkd\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.155986 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155984    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7dfe4c-4ecb-444f-950e-53ed7b72917b-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.156233 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.155993    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpmf\" (UniqueName: \"kubernetes.io/projected/4f6e78f4-f2df-43b5-874a-0095dd4a1f17-kube-api-access-jwpmf\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:14.567789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.567684    2564 generic.go:356] "Generic (PLEG): container finished" podID="ea4bd945-4e03-4f68-a850-3d45b597001d" containerID="e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f" exitCode=0
Jul 27 17:52:14.567789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.567723    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"
Jul 27 17:52:14.567789 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.567773    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" event={"ID":"ea4bd945-4e03-4f68-a850-3d45b597001d","Type":"ContainerDied","Data":"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"}
Jul 27 17:52:14.568087 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.567822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm" event={"ID":"ea4bd945-4e03-4f68-a850-3d45b597001d","Type":"ContainerDied","Data":"95f33d53ab9c5fd54ada16959d8178589f12f169fddc32b44e93d4170efe2883"}
Jul 27 17:52:14.568087 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.567836    2564 scope.go:117] "RemoveContainer" containerID="e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"
Jul 27 17:52:14.569302 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.569274    2564 generic.go:356] "Generic (PLEG): container finished" podID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" containerID="2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858" exitCode=0
Jul 27 17:52:14.569412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.569315    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"
Jul 27 17:52:14.569412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.569317    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" event={"ID":"3f7dfe4c-4ecb-444f-950e-53ed7b72917b","Type":"ContainerDied","Data":"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"}
Jul 27 17:52:14.569412 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.569367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh" event={"ID":"3f7dfe4c-4ecb-444f-950e-53ed7b72917b","Type":"ContainerDied","Data":"6ce0708698834472e94cf749c3f7b8d3d7ee77c3cda211f1f58fd66738979e97"}
Jul 27 17:52:14.570739 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.570721    2564 generic.go:356] "Generic (PLEG): container finished" podID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" containerID="74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7" exitCode=0
Jul 27 17:52:14.570840 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.570765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" event={"ID":"4f6e78f4-f2df-43b5-874a-0095dd4a1f17","Type":"ContainerDied","Data":"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"}
Jul 27 17:52:14.570840 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.570779    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"
Jul 27 17:52:14.570840 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.570799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4" event={"ID":"4f6e78f4-f2df-43b5-874a-0095dd4a1f17","Type":"ContainerDied","Data":"8f548a0e016cf981c0320e481e606a57a090e3a15947308ed84ef5762cf8c039"}
Jul 27 17:52:14.578235 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.578127    2564 scope.go:117] "RemoveContainer" containerID="e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"
Jul 27 17:52:14.578423 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:52:14.578395    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f\": container with ID starting with e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f not found: ID does not exist" containerID="e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"
Jul 27 17:52:14.578502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.578435    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f"} err="failed to get container status \"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f\": rpc error: code = NotFound desc = could not find container \"e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f\": container with ID starting with e70b20bf081f835cb802adc10223a352d1d1cbc8413fc51f79286d25999b268f not found: ID does not exist"
Jul 27 17:52:14.578502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.578455    2564 scope.go:117] "RemoveContainer" containerID="2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"
Jul 27 17:52:14.587013 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.586988    2564 scope.go:117] "RemoveContainer" containerID="2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"
Jul 27 17:52:14.587284 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:52:14.587262    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858\": container with ID starting with 2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858 not found: ID does not exist" containerID="2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"
Jul 27 17:52:14.587370 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.587294    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858"} err="failed to get container status \"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858\": rpc error: code = NotFound desc = could not find container \"2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858\": container with ID starting with 2e60c3d7fdb047d9c73ae5c6a14822f01416ba3d495a0400d1c84e143948b858 not found: ID does not exist"
Jul 27 17:52:14.587370 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.587316    2564 scope.go:117] "RemoveContainer" containerID="74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"
Jul 27 17:52:14.594605 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.594590    2564 scope.go:117] "RemoveContainer" containerID="74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"
Jul 27 17:52:14.594864 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:52:14.594818    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7\": container with ID starting with 74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7 not found: ID does not exist" containerID="74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"
Jul 27 17:52:14.594864 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.594838    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7"} err="failed to get container status \"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7\": rpc error: code = NotFound desc = could not find container \"74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7\": container with ID starting with 74e14b7ff40bdec48b4416095b69f8ac3a6707daaab9254a1772a39ab00a9bf7 not found: ID does not exist"
Jul 27 17:52:14.601055 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.601031    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:52:14.604746 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.604723    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-004618-66c7dc64c-f45gm"]
Jul 27 17:52:14.619310 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.619290    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:52:14.627069 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.626994    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-004618-5b94979b8b-fsfz4"]
Jul 27 17:52:14.641394 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.641352    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:52:14.643119 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.643098    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-004618-df4dc5588-qh2hh"]
Jul 27 17:52:14.658607 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.658578    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" path="/var/lib/kubelet/pods/3f7dfe4c-4ecb-444f-950e-53ed7b72917b/volumes"
Jul 27 17:52:14.659036 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.659018    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" path="/var/lib/kubelet/pods/4f6e78f4-f2df-43b5-874a-0095dd4a1f17/volumes"
Jul 27 17:52:14.659413 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:14.659396    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4bd945-4e03-4f68-a850-3d45b597001d" path="/var/lib/kubelet/pods/ea4bd945-4e03-4f68-a850-3d45b597001d/volumes"
Jul 27 17:52:16.510872 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:16.510840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:52:36.102695 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.102658    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:52:36.103382 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.102920    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" podUID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" containerName="maas-api" containerID="cri-o://f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2" gracePeriod=30
Jul 27 17:52:36.185815 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.185790    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:52:36.186089 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.186041    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" podUID="107421f9-d5a2-4332-bd03-2f01a391dea6" containerName="payload-processing" containerID="cri-o://a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4" gracePeriod=30
Jul 27 17:52:36.438364 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.438338    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:52:36.469386 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.469366    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:52:36.534746 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.534718    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs8kg\" (UniqueName: \"kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg\") pod \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") "
Jul 27 17:52:36.534866 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.534799    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume\") pod \"107421f9-d5a2-4332-bd03-2f01a391dea6\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") "
Jul 27 17:52:36.534866 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.534829    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca\") pod \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") "
Jul 27 17:52:36.534974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.534913    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") pod \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\" (UID: \"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04\") "
Jul 27 17:52:36.534974 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.534947    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88m8s\" (UniqueName: \"kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s\") pod \"107421f9-d5a2-4332-bd03-2f01a391dea6\" (UID: \"107421f9-d5a2-4332-bd03-2f01a391dea6\") "
Jul 27 17:52:36.535190 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.535163    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" (UID: "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:52:36.535311 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.535194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "107421f9-d5a2-4332-bd03-2f01a391dea6" (UID: "107421f9-d5a2-4332-bd03-2f01a391dea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:52:36.536917 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.536867    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg" (OuterVolumeSpecName: "kube-api-access-hs8kg") pod "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" (UID: "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04"). InnerVolumeSpecName "kube-api-access-hs8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:36.536917 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.536872    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" (UID: "1f19d9a6-2727-4d47-b4f9-bc5de4a52a04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:52:36.537090 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.536950    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s" (OuterVolumeSpecName: "kube-api-access-88m8s") pod "107421f9-d5a2-4332-bd03-2f01a391dea6" (UID: "107421f9-d5a2-4332-bd03-2f01a391dea6"). InnerVolumeSpecName "kube-api-access-88m8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:36.635514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.635489    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8kg\" (UniqueName: \"kubernetes.io/projected/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-kube-api-access-hs8kg\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:36.635514 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.635512    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107421f9-d5a2-4332-bd03-2f01a391dea6-plugins-config-volume\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:36.635655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.635522    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-openshift-service-ca\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:36.635655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.635531    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04-maas-api-tls\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:36.635655 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.635541    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88m8s\" (UniqueName: \"kubernetes.io/projected/107421f9-d5a2-4332-bd03-2f01a391dea6-kube-api-access-88m8s\") on node \"ip-10-0-129-84.ec2.internal\" DevicePath \"\""
Jul 27 17:52:36.671913 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.671873    2564 generic.go:356] "Generic (PLEG): container finished" podID="107421f9-d5a2-4332-bd03-2f01a391dea6" containerID="a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4" exitCode=0
Jul 27 17:52:36.672029 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.671941    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"
Jul 27 17:52:36.672029 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.671966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" event={"ID":"107421f9-d5a2-4332-bd03-2f01a391dea6","Type":"ContainerDied","Data":"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"}
Jul 27 17:52:36.672145 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.672029    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644" event={"ID":"107421f9-d5a2-4332-bd03-2f01a391dea6","Type":"ContainerDied","Data":"7814fb0e9e314164f22dd70572f44566963eebc342cae5eeb62567d85404e83c"}
Jul 27 17:52:36.672145 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.672064    2564 scope.go:117] "RemoveContainer" containerID="a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"
Jul 27 17:52:36.673287 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.673224    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" containerID="f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2" exitCode=0
Jul 27 17:52:36.673287 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.673269    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"
Jul 27 17:52:36.673421 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.673310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" event={"ID":"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04","Type":"ContainerDied","Data":"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"}
Jul 27 17:52:36.673421 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.673347    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc" event={"ID":"1f19d9a6-2727-4d47-b4f9-bc5de4a52a04","Type":"ContainerDied","Data":"03a398f41ebd5f40bb5ba208f66979d32aa518809d50a1d01f9c7b4084d22acf"}
Jul 27 17:52:36.686694 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.686674    2564 scope.go:117] "RemoveContainer" containerID="a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"
Jul 27 17:52:36.686976 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:52:36.686955    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4\": container with ID starting with a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4 not found: ID does not exist" containerID="a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"
Jul 27 17:52:36.687072 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.686988    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4"} err="failed to get container status \"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4\": rpc error: code = NotFound desc = could not find container \"a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4\": container with ID starting with a0b50d341336e8da08c209f210fa52208c73560a06af50291b76fba6234c54c4 not found: ID does not exist"
Jul 27 17:52:36.687072 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.687011    2564 scope.go:117] "RemoveContainer" containerID="f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"
Jul 27 17:52:36.692461 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.692437    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:52:36.694537 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.694520    2564 scope.go:117] "RemoveContainer" containerID="f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"
Jul 27 17:52:36.694807 ip-10-0-129-84 kubenswrapper[2564]: E0727 17:52:36.694786    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2\": container with ID starting with f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2 not found: ID does not exist" containerID="f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"
Jul 27 17:52:36.694870 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.694818    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2"} err="failed to get container status \"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2\": rpc error: code = NotFound desc = could not find container \"f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2\": container with ID starting with f17f8e05ac0a668de15e28e71342f7d81867a80cc51b1808c9b9103af1ea46f2 not found: ID does not exist"
Jul 27 17:52:36.696905 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.696868    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-509a5d-745f757454-hg644"]
Jul 27 17:52:36.711082 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.711060    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:52:36.716048 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:36.716030    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-509a5d-5bc94c974f-9whxc"]
Jul 27 17:52:38.657595 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:38.657560    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107421f9-d5a2-4332-bd03-2f01a391dea6" path="/var/lib/kubelet/pods/107421f9-d5a2-4332-bd03-2f01a391dea6/volumes"
Jul 27 17:52:38.657969 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:38.657930    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" path="/var/lib/kubelet/pods/1f19d9a6-2727-4d47-b4f9-bc5de4a52a04/volumes"
Jul 27 17:52:38.692794 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:38.692766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jx88v"]
Jul 27 17:52:54.656014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:54.655984    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-dcfj6_7766aaff-8379-4d13-8ff8-c1a38a03d2c4/copy-manifests/0.log"
Jul 27 17:52:54.663372 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:54.663343    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-dcfj6_7766aaff-8379-4d13-8ff8-c1a38a03d2c4/manager/0.log"
Jul 27 17:52:57.454130 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:57.454098    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wzd6l_81dcacb1-9a9c-468d-be3b-a89d11b50a37/manager/0.log"
Jul 27 17:52:57.564895 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:57.564851    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jx88v_21e31e58-e858-420f-beba-75d6d9d0c79d/limitador/0.log"
Jul 27 17:52:58.406910 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:58.406866    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-jpfpv_d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6/kube-auth-proxy/0.log"
Jul 27 17:52:59.845426 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:52:59.845391    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-hcczl_3eda460f-7cc5-46c4-9117-4f85e98a1df4/main/0.log"
Jul 27 17:53:05.845800 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.845771    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tn87g/must-gather-ft7pq"]
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846085    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4bd945-4e03-4f68-a850-3d45b597001d" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846096    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4bd945-4e03-4f68-a850-3d45b597001d" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846104    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846110    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846116    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" containerName="payload-pre-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846121    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" containerName="payload-pre-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846129    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15de0f11-7157-430c-889b-5649c416ab1a" containerName="payload-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846135    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15de0f11-7157-430c-889b-5649c416ab1a" containerName="payload-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846142    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" containerName="payload-pre-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846148    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" containerName="payload-pre-processing"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846155    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" containerName="main"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846160    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" containerName="main"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846169    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31045e1-5a1e-4808-85be-901cc449259c" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846174    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31045e1-5a1e-4808-85be-901cc449259c" containerName="maas-api"
Jul 27 17:53:05.846173 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846181    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107421f9-d5a2-4332-bd03-2f01a391dea6" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846187    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="107421f9-d5a2-4332-bd03-2f01a391dea6" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846196    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846201    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846207    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1c26c8-8d6f-4369-9644-c2389794d24e" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846212    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1c26c8-8d6f-4369-9644-c2389794d24e" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846224    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846229    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846272    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6e78f4-f2df-43b5-874a-0095dd4a1f17" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846279    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8ba8dcc-ca02-4ac2-a589-3c6e2276cb0a" containerName="payload-pre-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846301    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31045e1-5a1e-4808-85be-901cc449259c" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846308    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7dfe4c-4ecb-444f-950e-53ed7b72917b" containerName="payload-pre-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846315    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4bd945-4e03-4f68-a850-3d45b597001d" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846322    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f19d9a6-2727-4d47-b4f9-bc5de4a52a04" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846327    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1c26c8-8d6f-4369-9644-c2389794d24e" containerName="maas-api"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846332    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="107421f9-d5a2-4332-bd03-2f01a391dea6" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846338    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86808ed5-0fe5-41d1-9f6f-d8cbbea0b5a4" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846345    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="15de0f11-7157-430c-889b-5649c416ab1a" containerName="payload-processing"
Jul 27 17:53:05.846636 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.846350    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ae4209-2bd3-40d8-9a1c-5aafe2ecb396" containerName="main"
Jul 27 17:53:05.849491 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.849475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:05.852560 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.852536    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tn87g\"/\"default-dockercfg-78w79\""
Jul 27 17:53:05.852653 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.852585    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn87g\"/\"kube-root-ca.crt\""
Jul 27 17:53:05.852973 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.852960    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tn87g\"/\"openshift-service-ca.crt\""
Jul 27 17:53:05.871070 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.871049    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/must-gather-ft7pq"]
Jul 27 17:53:05.960209 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.960188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf572\" (UniqueName: \"kubernetes.io/projected/0fecd9e7-b5cd-479b-afcc-d65827492a6d-kube-api-access-pf572\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:05.960319 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:05.960221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fecd9e7-b5cd-479b-afcc-d65827492a6d-must-gather-output\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.061230 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.061205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf572\" (UniqueName: \"kubernetes.io/projected/0fecd9e7-b5cd-479b-afcc-d65827492a6d-kube-api-access-pf572\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.061320 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.061241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fecd9e7-b5cd-479b-afcc-d65827492a6d-must-gather-output\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.061553 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.061537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fecd9e7-b5cd-479b-afcc-d65827492a6d-must-gather-output\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.072210 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.072191    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf572\" (UniqueName: \"kubernetes.io/projected/0fecd9e7-b5cd-479b-afcc-d65827492a6d-kube-api-access-pf572\") pod \"must-gather-ft7pq\" (UID: \"0fecd9e7-b5cd-479b-afcc-d65827492a6d\") " pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.158415 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.158396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/must-gather-ft7pq"
Jul 27 17:53:06.284802 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.284780    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/must-gather-ft7pq"]
Jul 27 17:53:06.286855 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:53:06.286830    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fecd9e7_b5cd_479b_afcc_d65827492a6d.slice/crio-9b0d6c8bc41284dd1edb2e7e1e51f21328287bb7dc790f7494b73f0106531f78 WatchSource:0}: Error finding container 9b0d6c8bc41284dd1edb2e7e1e51f21328287bb7dc790f7494b73f0106531f78: Status 404 returned error can't find the container with id 9b0d6c8bc41284dd1edb2e7e1e51f21328287bb7dc790f7494b73f0106531f78
Jul 27 17:53:06.803871 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:06.803841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/must-gather-ft7pq" event={"ID":"0fecd9e7-b5cd-479b-afcc-d65827492a6d","Type":"ContainerStarted","Data":"9b0d6c8bc41284dd1edb2e7e1e51f21328287bb7dc790f7494b73f0106531f78"}
Jul 27 17:53:07.810969 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:07.810932    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/must-gather-ft7pq" event={"ID":"0fecd9e7-b5cd-479b-afcc-d65827492a6d","Type":"ContainerStarted","Data":"74345edc2cac7b7337256a22bc0f3a0b635a78a4e0c5d6650721d41e87dd6cec"}
Jul 27 17:53:07.810969 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:07.810975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/must-gather-ft7pq" event={"ID":"0fecd9e7-b5cd-479b-afcc-d65827492a6d","Type":"ContainerStarted","Data":"c702b73b0d93d440309c07a8e762c217be5de645f505ca089a49ae298d8ec78e"}
Jul 27 17:53:07.832415 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:07.832367    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tn87g/must-gather-ft7pq" podStartSLOduration=2.17256988 podStartE2EDuration="2.832352063s" totalImagesPullingTime="659.782183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:07.831011842 +0000 UTC m=+3407.663376819" watchObservedRunningTime="2026-07-27 17:53:07.832352063 +0000 UTC m=+3407.664717039"
Jul 27 17:53:08.636457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:08.636384    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8w67f_6762120c-be3f-4667-adc9-a6a118e1ae83/global-pull-secret-syncer/0.log"
Jul 27 17:53:08.841592 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:08.841565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ljms5_48c5ca51-b05b-45c3-bece-1db371d1e4f1/konnectivity-agent/0.log"
Jul 27 17:53:08.938462 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:08.938436    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-84.ec2.internal_fb729aa8976fe5911d6204f16973d9d7/haproxy/0.log"
Jul 27 17:53:14.485304 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:14.485274    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wzd6l_81dcacb1-9a9c-468d-be3b-a89d11b50a37/manager/0.log"
Jul 27 17:53:14.574496 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:14.574442    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jx88v_21e31e58-e858-420f-beba-75d6d9d0c79d/limitador/0.log"
Jul 27 17:53:16.830685 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:16.830654    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-kt6d2_6ca26a8a-542c-40f6-9af6-d06625d47632/cluster-monitoring-operator/0.log"
Jul 27 17:53:17.130457 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.130426    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/node-exporter/0.log"
Jul 27 17:53:17.200369 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.200338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/kube-rbac-proxy/0.log"
Jul 27 17:53:17.273014 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.272989    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ppmn9_513eb75f-69f6-4367-897d-9dad88c76bdf/init-textfile/0.log"
Jul 27 17:53:17.781810 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.781781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/prometheus/0.log"
Jul 27 17:53:17.856258 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.856219    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/config-reloader/0.log"
Jul 27 17:53:17.929085 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.929060    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/thanos-sidecar/0.log"
Jul 27 17:53:17.990689 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:17.990640    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/kube-rbac-proxy-web/0.log"
Jul 27 17:53:18.076978 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.076903    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/kube-rbac-proxy/0.log"
Jul 27 17:53:18.133637 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.133613    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/kube-rbac-proxy-thanos/0.log"
Jul 27 17:53:18.195810 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.195783    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_26f8f641-4932-468b-9933-504c96e7e097/init-config-reloader/0.log"
Jul 27 17:53:18.257310 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.257211    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kmnwq_4e1d4173-de1d-4976-abfa-1905ffc4333e/prometheus-operator/0.log"
Jul 27 17:53:18.323375 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.323348    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kmnwq_4e1d4173-de1d-4976-abfa-1905ffc4333e/kube-rbac-proxy/0.log"
Jul 27 17:53:18.396079 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.396047    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"]
Jul 27 17:53:18.403669 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.403639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.411812 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.411754    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-7ft5s_6b9804e1-6ef0-4a2f-91c7-feef292925d2/prometheus-operator-admission-webhook/0.log"
Jul 27 17:53:18.436628 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.436596    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"]
Jul 27 17:53:18.478911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.477949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-proc\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.478911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.478024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-sys\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.478911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.478054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-lib-modules\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.478911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.478080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-525ps\" (UniqueName: \"kubernetes.io/projected/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-kube-api-access-525ps\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.478911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.478143    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-podres\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.579604 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-sys\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.579818 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579629    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-lib-modules\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.579818 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-525ps\" (UniqueName: \"kubernetes.io/projected/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-kube-api-access-525ps\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.579818 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579738    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-podres\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.579818 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-proc\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.580074 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579935    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-proc\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.580074 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.579992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-sys\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.580186 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.580102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-lib-modules\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.580497 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.580469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-podres\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.592035 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.592007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-525ps\" (UniqueName: \"kubernetes.io/projected/c47ce4f6-e820-4614-a3c5-371ea3dbbea2-kube-api-access-525ps\") pod \"perf-node-gather-daemonset-7r2tw\" (UID: \"c47ce4f6-e820-4614-a3c5-371ea3dbbea2\") " pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.672328 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.672253    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/thanos-query/0.log"
Jul 27 17:53:18.715303 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.715272    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-web/0.log"
Jul 27 17:53:18.716333 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.716304    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:18.775966 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.775935    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy/0.log"
Jul 27 17:53:18.842904 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.842853    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/prom-label-proxy/0.log"
Jul 27 17:53:18.893146 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.893102    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"]
Jul 27 17:53:18.894148 ip-10-0-129-84 kubenswrapper[2564]: W0727 17:53:18.894120    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc47ce4f6_e820_4614_a3c5_371ea3dbbea2.slice/crio-da848fd6bbc1dae6c6f7c9260de9b2a957b5fabcc7774a0ecc05b2e7186dc634 WatchSource:0}: Error finding container da848fd6bbc1dae6c6f7c9260de9b2a957b5fabcc7774a0ecc05b2e7186dc634: Status 404 returned error can't find the container with id da848fd6bbc1dae6c6f7c9260de9b2a957b5fabcc7774a0ecc05b2e7186dc634
Jul 27 17:53:18.920375 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.920344    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-rules/0.log"
Jul 27 17:53:18.978610 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:18.978582    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867779b548-7hf7r_d349f194-1f47-4292-9b5e-d8377de5b469/kube-rbac-proxy-metrics/0.log"
Jul 27 17:53:19.884183 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:19.883280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw" event={"ID":"c47ce4f6-e820-4614-a3c5-371ea3dbbea2","Type":"ContainerStarted","Data":"b14a44aeb5a071af73cebd964a73e2a94ae837e59f7e8f8de55a8afd09b8337e"}
Jul 27 17:53:19.884183 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:19.883343    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw" event={"ID":"c47ce4f6-e820-4614-a3c5-371ea3dbbea2","Type":"ContainerStarted","Data":"da848fd6bbc1dae6c6f7c9260de9b2a957b5fabcc7774a0ecc05b2e7186dc634"}
Jul 27 17:53:19.884183 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:19.884122    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:19.917502 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:19.917452    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw" podStartSLOduration=1.9174357899999999 podStartE2EDuration="1.91743579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:19.916039492 +0000 UTC m=+3419.748404469" watchObservedRunningTime="2026-07-27 17:53:19.91743579 +0000 UTC m=+3419.749800744"
Jul 27 17:53:20.831074 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:20.831049    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/2.log"
Jul 27 17:53:20.835363 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:20.835339    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5w7vz_30c41d72-0d82-4f03-a4bd-19dd758244ae/console-operator/3.log"
Jul 27 17:53:23.458083 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:23.458059    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86hzs_3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f/dns/0.log"
Jul 27 17:53:23.492635 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:23.492610    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86hzs_3e40ad66-289f-4d3b-bf13-1a1a1ef1f78f/kube-rbac-proxy/0.log"
Jul 27 17:53:23.588928 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:23.588899    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7mpd_59367e1f-f4c4-4c3b-8180-0068fd1cd54f/dns-node-resolver/0.log"
Jul 27 17:53:24.382673 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:24.382630    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4bhv6_5b0005a1-5a7a-48df-b38a-cc69f15efd96/node-ca/0.log"
Jul 27 17:53:25.841619 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:25.841558    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-jpfpv_d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6/kube-auth-proxy/0.log"
Jul 27 17:53:26.904997 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:26.904973    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tn87g/perf-node-gather-daemonset-7r2tw"
Jul 27 17:53:27.120994 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:27.120968    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7srl7_bf4adb49-ee35-4f55-b2b6-36b2510f5080/serve-healthcheck-canary/0.log"
Jul 27 17:53:27.985342 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:27.985312    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkt7j_c95f73ba-bcd4-46d1-a5d1-6cb8a4300188/kube-rbac-proxy/0.log"
Jul 27 17:53:28.013126 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:28.013096    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkt7j_c95f73ba-bcd4-46d1-a5d1-6cb8a4300188/exporter/0.log"
Jul 27 17:53:28.043652 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:28.043625    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mkt7j_c95f73ba-bcd4-46d1-a5d1-6cb8a4300188/extractor/0.log"
Jul 27 17:53:28.642833 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:28.642745    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-jpfpv_d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6/kube-auth-proxy/0.log"
Jul 27 17:53:30.454276 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:30.454228    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc796b767-jpfpv_d3a7e101-87da-40ce-b3f5-7eaed4b6f0a6/kube-auth-proxy/0.log"
Jul 27 17:53:31.423113 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:31.423069    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-dcfj6_7766aaff-8379-4d13-8ff8-c1a38a03d2c4/manager/0.log"
Jul 27 17:53:31.461546 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:31.461523    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565c484996-dcfj6_7766aaff-8379-4d13-8ff8-c1a38a03d2c4/copy-manifests/0.log"
Jul 27 17:53:32.882911 ip-10-0-129-84 kubenswrapper[2564]: I0727 17:53:32.882886    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b56999664-dstbd_5715a615-4359-43f5-a41c-54dd9f6c95b4/manager/0.log"