Sep 10 13:26:37.735545 ip-10-0-133-48 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 13:26:37.735555 ip-10-0-133-48 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 13:26:37.735563 ip-10-0-133-48 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 13:26:37.735714 ip-10-0-133-48 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 13:26:47.984731 ip-10-0-133-48 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 13:26:47.984749 ip-10-0-133-48 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6a2cf0a728b640aeadf7e0e72633602f --
Sep 10 13:28:35.437801 ip-10-0-133-48 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 13:28:35.862002 ip-10-0-133-48 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:28:35.862002 ip-10-0-133-48 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 13:28:35.862002 ip-10-0-133-48 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:28:35.862002 ip-10-0-133-48 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 13:28:35.862002 ip-10-0-133-48 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:28:35.863100 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.862978    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 13:28:35.867284 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867266    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.867284 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867284    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.867284 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867291    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867296    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867301    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867305    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867309    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867314    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867317    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867321    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867326    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867330    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867334    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867340    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867346    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867350    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867354    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867358    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867362    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867366    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867370    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.867479 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867374    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867378    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867382    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867393    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867397    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867401    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867405    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867409    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867414    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867418    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867422    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867428    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867433    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867436    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867440    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867446    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867450    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867455    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867461    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.868276 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867465    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867472    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867478    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867482    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867487    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867491    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867495    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867499    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867520    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867525    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867530    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867534    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867538    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867542    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867547    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867551    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867556    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867560    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867563    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867568    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.869089 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867572    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867576    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867580    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867585    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867589    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867593    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867598    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867601    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867606    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867611    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867615    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867621    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867626    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867630    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867634    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867638    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867642    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867648    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867653    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867657    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.869945 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867662    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867666    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867671    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867675    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867679    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867683    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867687    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867691    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867695    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867699    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867703    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867708    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867712    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867716    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867721    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867724    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867729    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867733    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.867737    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868882    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.870815 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868892    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868898    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868902    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868907    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868912    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868916    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868920    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868925    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868929    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868933    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868939    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868946    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868952    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868956    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868960    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868964    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868968    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868973    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868977    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.871723 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868981    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868985    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868989    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868993    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.868998    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869002    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869006    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869010    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869015    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869019    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869023    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869027    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869031    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869036    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869040    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869044    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869051    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869056    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869063    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869068    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.872359 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869073    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869077    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869081    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869085    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869089    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869093    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869097    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869101    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869105    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869108    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869112    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869116    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869120    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869124    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869129    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869134    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869138    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869143    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869148    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869152    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.872939 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869156    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869160    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869164    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869168    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869173    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869178    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869182    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869187    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869191    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869195    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869199    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869204    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869209    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869215    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869219    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869223    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869227    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869231    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869236    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869239    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.873524 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869246    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869252    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869257    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869262    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869267    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869271    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869276    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869280    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869284    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869289    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869293    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869298    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869302    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869306    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869310    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869314    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869318    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869322    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869326    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869928    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.874045 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869936    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869940    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869944    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869949    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869953    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869957    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869963    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869967    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869975    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869980    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869984    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869988    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869992    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.869997    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870001    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870005    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870009    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870013    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870018    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870022    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.874569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870026    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870030    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870036    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870043    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870047    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870051    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870055    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870060    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870064    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870069    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870073    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870077    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870082    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870086    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870098    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870102    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870108    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870116    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870122    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.875184 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870127    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870133    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870138    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870143    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870147    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870151    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870155    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870160    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870164    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870168    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870175    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870181    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870185    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870189    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870194    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870198    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870202    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870206    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870210    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870215    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.875687 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870219    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870223    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870227    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870231    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870238    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870242    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870246    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870250    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870255    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870259    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870264    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870268    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870272    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870278    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870282    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870286    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870290    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870294    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870298    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870304    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.876340 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870308    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870312    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870317    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870321    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870326    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870330    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870334    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870338    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870342    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870347    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870351    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870354    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870358    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870362    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870367    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870371    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870375    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870379    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.870383    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870487    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 13:28:35.876902 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870566    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870583    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870590    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870604    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870609    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870617    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870625    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870630    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870634    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870641    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870646    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870653    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870658    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870662    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870668    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870672    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870702    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870713    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870718    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870723    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870729    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870734    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870741    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 13:28:35.877464 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870746    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870750    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870756    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870761    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870765    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870770    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870775    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870780    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870786    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870791    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870796    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870801    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870806    2577 flags.go:64] FLAG: --enable-server="true"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870812    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870821    2577 flags.go:64] FLAG: --event-burst="100"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870826    2577 flags.go:64] FLAG: --event-qps="50"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870830    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870835    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870840    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870846    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870853    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870858    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870862    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870867    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870871    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 13:28:35.878052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870876    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870881    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870886    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870891    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870896    2577 flags.go:64] FLAG: --feature-gates=""
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870906    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870911    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870916    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870921    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870926    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870931    2577 flags.go:64] FLAG: --help="false"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870935    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870940    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870945    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870950    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870956    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870962    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870966    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870971    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870975    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870981    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870990    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870995    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.870999    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 10 13:28:35.878721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871004    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871009    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871014    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871020    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871025    2577 flags.go:64] FLAG: --lock-file=""
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871030    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871034    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871039    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871048    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871053    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871057    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871062    2577 flags.go:64] FLAG: --logging-format="text"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871066    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871072    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871077    2577 flags.go:64] FLAG: --manifest-url=""
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871082    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871089    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871093    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871100    2577 flags.go:64] FLAG: --max-pods="110"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871105    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871110    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871114    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871119    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871124    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871129    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 13:28:35.879308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871134    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871146    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871151    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871156    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871162    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871168    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871177    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871181    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871186    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871191    2577 flags.go:64] FLAG: --port="10250"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871198    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871202    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f38561a5dd986112"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871208    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871213    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871217    2577 flags.go:64] FLAG: --register-node="true"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871222    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871228    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871233    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871238    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871242    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871248    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871253    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871257    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871262    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871267    2577 flags.go:64] FLAG: --runonce="false"
Sep 10 13:28:35.879973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871272    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871277    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871282    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871287    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871291    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871296    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871301    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871306    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871310    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871315    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871320    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871325    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871331    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871337    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871341    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871350    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871355    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871361    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871367    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871372    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871377    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871381    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871386    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871391    2577 flags.go:64] FLAG: --v="2"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871397    2577 flags.go:64] FLAG: --version="false"
Sep 10 13:28:35.880724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871403    2577 flags.go:64] FLAG: --vmodule=""
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871409    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.871414    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871617    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871625    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871630    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871635    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871639    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871643    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871648    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871652    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871656    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871660    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871664    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871668    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871673    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871677    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871681    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871685    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871691    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.881331 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871695    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871700    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871704    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871709    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871715    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871719    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871723    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871726    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871730    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871734    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871738    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871745    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871751    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871756    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871760    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871765    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871769    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871773    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871777    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.881843 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871782    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871787    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871791    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871795    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871799    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871803    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871807    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871811    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871815    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871821    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871827    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871832    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871838    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871843    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871847    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871852    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871857    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871862    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871867    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.882320 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871871    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871876    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871880    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871884    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871888    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871892    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871896    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871900    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871904    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871909    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871913    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871917    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871921    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871925    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871930    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871934    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871938    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871942    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871949    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871954    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.882805 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871959    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871964    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871968    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871973    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871978    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871983    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871988    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871992    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.871996    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872000    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872006    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872011    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872015    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872019    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872023    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872027    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872031    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872035    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872039    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872042    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.883300 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872047    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872051    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872055    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.872060    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.873061    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.881891    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882020    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882042    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882048    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882114    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882120    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882123    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882126    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882129    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.883813 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882132    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882135    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882138    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882140    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882143    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882145    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882148    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882150    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882153    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882156    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882158    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882160    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882163    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882166    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882168    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882171    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882173    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882176    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882179    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882181    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.884156 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882184    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882186    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882190    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882194    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882199    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882202    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882205    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882208    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882211    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882214    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882217    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882219    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882222    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882225    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882227    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882230    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882232    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882235    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882237    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.884760 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882240    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882242    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882245    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882247    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882250    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882253    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882255    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882258    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882260    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882263    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882265    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882267    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882270    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882272    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882275    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882278    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882280    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882284    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882287    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882290    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.885215 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882293    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882295    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882297    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882300    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882302    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882305    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882307    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882310    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882312    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882315    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882317    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882320    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882323    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882325    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882328    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882331    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882333    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882336    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882339    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.885717 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882341    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882344    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882348    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882351    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882354    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882357    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882360    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882363    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882365    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882369    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882373    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882377    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882380    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882383    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882386    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.886182 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882388    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882393    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882518    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882523    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882526    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882529    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882532    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882535    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882538    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882540    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882543    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882545    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882548    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882551    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882553    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:28:35.886568 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882556    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882558    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882561    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882563    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882566    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882568    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882571    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882574    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882576    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882579    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882581    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882584    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882587    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882589    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882592    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882595    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882598    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882600    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882603    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882605    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:28:35.886932 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882608    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882610    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882613    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882617    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882621    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882624    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882627    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882630    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882632    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882635    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882637    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882640    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882642    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882645    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882647    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882650    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882652    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882655    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882657    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:28:35.887416 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882660    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882662    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882665    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882668    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882670    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882673    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882676    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882680    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882683    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882686    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882688    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882691    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882693    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882696    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882698    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882701    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882703    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882706    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882708    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:28:35.887933 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882710    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882713    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882716    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882718    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882721    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882723    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882726    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882729    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882732    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882734    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882737    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882740    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882743    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882745    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882748    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882750    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882753    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882757    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882760    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:28:35.888403 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882763    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882766    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882769    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882772    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882775    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882777    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882780    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882783    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:35.882786    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882790    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.882906    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.885265    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.887032    2577 server.go:1019] "Starting client certificate rotation"
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.887129    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:28:35.888881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.887169    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:28:35.893620 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.893601    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:28:35.895778 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.895762    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:28:35.907358 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.907333    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 10 13:28:35.910793 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.910778    2577 log.go:25] "Validated CRI v1 image API"
Sep 10 13:28:35.911577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.911556    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 13:28:35.914115 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.914086    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c0750084-9fc6-4b4e-9ccf-aa7af69f135f:/dev/nvme0n1p4 e3b682c8-fe95-47a1-b4f0-b23888716253:/dev/nvme0n1p3]
Sep 10 13:28:35.914172 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.914115    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 13:28:35.917711 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.917693    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:28:35.920148 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.920042    2577 manager.go:216] Machine: {Timestamp:2026-09-10 13:28:35.918687943 +0000 UTC m=+0.373563097 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099897 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f74aecabaf826a10ddc4c7a4ecfc0 SystemUUID:ec2f74ae-caba-f826-a10d-dc4c7a4ecfc0 BootID:6a2cf0a7-28b6-40ae-adf7-e0e72633602f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:d6:53:91:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:d6:53:91:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:e1:f1:2b:b0:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 13:28:35.920148 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.920138    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 13:28:35.920315 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.920215    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 13:28:35.922986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.922960    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 13:28:35.923161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.922990    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 13:28:35.924406 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.924394    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 13:28:35.924459 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.924411    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 13:28:35.924459 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.924423    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:28:35.925485 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.925472    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:28:35.927407 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.927395    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 13:28:35.931564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931550    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:28:35.931696 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931685    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 13:28:35.931757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931723    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 13:28:35.931757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931739    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 13:28:35.931848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931759    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 13:28:35.931848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931776    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 10 13:28:35.931848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.931788    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 13:28:35.932152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.932139    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:28:35.932211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.932162    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:28:35.935798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.935776    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 13:28:35.938878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.938864    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 13:28:35.938937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.938893    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 13:28:35.939057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939047    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 13:28:35.939087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939062    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 13:28:35.939087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939067    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 13:28:35.939087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939073    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 13:28:35.939087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939080    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 13:28:35.939087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939086    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 13:28:35.939814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939804    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 13:28:35.939814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939814    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 13:28:35.939873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939821    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 13:28:35.939873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939827    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 13:28:35.939873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939834    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 13:28:35.939873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.939842    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 13:28:35.941301 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.941289    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 13:28:35.941301 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.941301    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 13:28:35.942194 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.942172    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:28:35.942194 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.942172    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:28:35.945447 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.945434    2577 server.go:1268] "Started kubelet"
Sep 10 13:28:35.945569 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.945544    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 13:28:35.945652 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.945534    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 13:28:35.945698 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.945670    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 13:28:35.946381 ip-10-0-133-48 systemd[1]: Started Kubernetes Kubelet.
Sep 10 13:28:35.947068 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.947052    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 13:28:35.947521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.947492    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 13:28:35.949527 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.948668    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78v7x"
Sep 10 13:28:35.949846 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.949830    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 13:28:35.950961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.950886    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:35.951851 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.950790    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f76f2450  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.945407568 +0000 UTC m=+0.400282722,LastTimestamp:2026-09-10 13:28:35.945407568 +0000 UTC m=+0.400282722,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:35.952689 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.952672    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 13:28:35.953329 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.953313    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 13:28:35.953624 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.953605    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-48.ec2.internal\" not found"
Sep 10 13:28:35.954050 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954035    2577 factory.go:55] Registering systemd factory
Sep 10 13:28:35.954050 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954050    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 10 13:28:35.954194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954175    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 13:28:35.954194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954186    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 13:28:35.954284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954202    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 13:28:35.954284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954257    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 13:28:35.954284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954263    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 13:28:35.954403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954314    2577 factory.go:153] Registering CRI-O factory
Sep 10 13:28:35.954403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954333    2577 factory.go:223] Registration of the crio container factory successfully
Sep 10 13:28:35.954403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954397    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 13:28:35.954553 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954431    2577 factory.go:103] Registering Raw factory
Sep 10 13:28:35.954553 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954446    2577 manager.go:1196] Started watching for new ooms in manager
Sep 10 13:28:35.954856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.954841    2577 manager.go:349] Starting recovery of all containers
Sep 10 13:28:35.961118 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.961082    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:28:35.961243 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.961208    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 13:28:35.963563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.963526    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 13:28:35.965016 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.965000    2577 manager.go:354] Recovery completed
Sep 10 13:28:35.969757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.969744    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:35.971399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.971385    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:35.971454 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.971413    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:35.971454 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.971423    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:35.972001 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.971978    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 13:28:35.972001 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.971994    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 13:28:35.972160 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.972011    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:28:35.972160 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.972009    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:35.972752 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.972739    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:35.972817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.972767    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:35.972817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.972781    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:35.973053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.973041    2577 policy_none.go:49] "None policy: Start"
Sep 10 13:28:35.973108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.973056    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 13:28:35.973108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.973066    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 13:28:35.973408 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.973335    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:35.974112 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.974098    2577 policy_none.go:47] "Start"
Sep 10 13:28:35.982014 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.981923    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:35.991837 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.991746    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:35.999212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.999193    2577 manager.go:337] "Starting Device Plugin manager"
Sep 10 13:28:35.999321 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:35.999260    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 13:28:35.999321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.999273    2577 server.go:87] "Starting device plugin registration server"
Sep 10 13:28:35.999634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.999619    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 13:28:35.999699 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.999635    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 13:28:35.999796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:35.999783    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 13:28:36.000372 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.000297    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 13:28:36.000372 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.000355    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-48.ec2.internal\" not found"
Sep 10 13:28:36.000372 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.000303    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 13:28:36.000372 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.000374    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 13:28:36.000868 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.000775    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:35.972750842 +0000 UTC m=+0.427625996,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.009483 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.009395    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:35.972773775 +0000 UTC m=+0.427648931,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.029750 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.021679    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:35.972788156 +0000 UTC m=+0.427663309,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.030641 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.030567    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0face1570  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:36.001961328 +0000 UTC m=+0.456836473,LastTimestamp:2026-09-10 13:28:36.001961328 +0000 UTC m=+0.456836473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.056913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.056888    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 13:28:36.057013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.056938    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 13:28:36.057013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.056958    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 13:28:36.057076 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.056998    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 13:28:36.069277 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.069250    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:28:36.100330 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.100310    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.101401 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.101372    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.101523 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.101407    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.101523 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.101421    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.101523 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.101451    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.110443 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.110357    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.101389008 +0000 UTC m=+0.556264168,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.119807 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.119671    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.101413737 +0000 UTC m=+0.556288893,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.130314 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.130295    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.130415 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.130343    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:36.101426658 +0000 UTC m=+0.556301813,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.157703 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.157678    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"]
Sep 10 13:28:36.157752 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.157745    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.159059 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.159042    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.159157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.159075    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.159157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.159089    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.161086 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161074    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.161202 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.161237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161216    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.161825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161809    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.161891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161839    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.161891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161852    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.161891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161811    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.161979 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161910    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.161979 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.161923    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.163848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.163835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.163912 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.163860    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.164604 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.164582    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.164685 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.164612    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.164685 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.164625    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.169636 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.169570    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.159057814 +0000 UTC m=+0.613932968,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.169636 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.169631    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 10 13:28:36.178156 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.178100    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.159082195 +0000 UTC m=+0.613957348,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.190200 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.190178    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.194066 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.194010    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:36.159093417 +0000 UTC m=+0.613968571,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.194612 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.194596    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.202035 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.201964    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.161828329 +0000 UTC m=+0.616703484,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.209772 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.209718    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.161845069 +0000 UTC m=+0.616720222,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.218731 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.218682    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:36.161855492 +0000 UTC m=+0.616730646,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.228590 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.228534    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.161895634 +0000 UTC m=+0.616770991,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.237825 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.237765    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.161916311 +0000 UTC m=+0.616791466,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.247337 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.247279    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:36.161926966 +0000 UTC m=+0.616802120,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.255140 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.255075    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.164596678 +0000 UTC m=+0.619471837,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.255623 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.255597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.255719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.255626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.255719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.255662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13f9881a5f465bf70dd944de0203fca5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-48.ec2.internal\" (UID: \"13f9881a5f465bf70dd944de0203fca5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.263863 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.263806    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.164618777 +0000 UTC m=+0.619493935,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.272855 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.272800    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc2c20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.97142736 +0000 UTC m=+0.426302514,LastTimestamp:2026-09-10 13:28:36.164631054 +0000 UTC m=+0.619506210,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.330955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.330934    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.332835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.332818    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.332914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.332848    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.332914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.332864    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.332914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.332897    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.341311 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.341217    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fbc1ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971400138 +0000 UTC m=+0.426275293,LastTimestamp:2026-09-10 13:28:36.332834302 +0000 UTC m=+0.787709456,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.349687 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.349614    2577 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-48.ec2.internal.18d3f8b0f8fc088b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-48.ec2.internal,UID:ip-10-0-133-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:35.971418251 +0000 UTC m=+0.426293405,LastTimestamp:2026-09-10 13:28:36.332855762 +0000 UTC m=+0.787730916,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:36.349874 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.349855    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.355962 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.355941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.356036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.355978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.356036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.356006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13f9881a5f465bf70dd944de0203fca5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-48.ec2.internal\" (UID: \"13f9881a5f465bf70dd944de0203fca5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.356132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.356050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.356132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.356069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13f9881a5f465bf70dd944de0203fca5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-48.ec2.internal\" (UID: \"13f9881a5f465bf70dd944de0203fca5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.356132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.356082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/159008f3edd5171bb1bb440b936be3a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal\" (UID: \"159008f3edd5171bb1bb440b936be3a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.491734 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.491664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.497260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.497091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.572706 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.572666    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Sep 10 13:28:36.750335 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.750307    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:36.751399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.751383    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:36.751471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.751418    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:36.751471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.751431    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:36.751471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.751460    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.777441 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.777407    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:36.888105 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.888064    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:28:36.907928 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:36.907887    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:28:36.962333 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:36.962300    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:37.044472 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.044439    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:37.045884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.045847    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:37.046005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.045895    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:37.046005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.045911    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:37.102173 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.102142    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:28:37.135272 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:37.135077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159008f3edd5171bb1bb440b936be3a3.slice/crio-a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b WatchSource:0}: Error finding container a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b: Status 404 returned error can't find the container with id a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b
Sep 10 13:28:37.135544 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:37.135502    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f9881a5f465bf70dd944de0203fca5.slice/crio-eb93ef347c7233d330abc46ebcadb495a0cf42231ccc9bd8463d89bcef70a32f WatchSource:0}: Error finding container eb93ef347c7233d330abc46ebcadb495a0cf42231ccc9bd8463d89bcef70a32f: Status 404 returned error can't find the container with id eb93ef347c7233d330abc46ebcadb495a0cf42231ccc9bd8463d89bcef70a32f
Sep 10 13:28:37.140605 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.140580    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:28:37.156719 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.156646    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-48.ec2.internal.18d3f8b13eaf9f5b  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-48.ec2.internal,UID:13f9881a5f465bf70dd944de0203fca5,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\",Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:37.140815707 +0000 UTC m=+1.595690854,LastTimestamp:2026-09-10 13:28:37.140815707 +0000 UTC m=+1.595690854,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:37.172606 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.172512    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal.18d3f8b13eb0b454  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal,UID:159008f3edd5171bb1bb440b936be3a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\",Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:37.140886612 +0000 UTC m=+1.595761754,LastTimestamp:2026-09-10 13:28:37.140886612 +0000 UTC m=+1.595761754,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:37.382852 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.382762    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Sep 10 13:28:37.537246 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.537206    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:28:37.577858 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.577827    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:37.578934 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.578906    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:37.579079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.578950    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:37.579079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.578968    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:37.579079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.579016    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:37.599348 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:37.599311    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:37.963453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:37.963416    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:38.062416 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:38.062353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal" event={"ID":"13f9881a5f465bf70dd944de0203fca5","Type":"ContainerStarted","Data":"eb93ef347c7233d330abc46ebcadb495a0cf42231ccc9bd8463d89bcef70a32f"}
Sep 10 13:28:38.065546 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:38.065516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerStarted","Data":"a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b"}
Sep 10 13:28:38.602053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:38.602014    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:28:38.962898 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:38.962824    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:38.992425 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:38.992385    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Sep 10 13:28:39.082611 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.082515    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal.18d3f8b1b1abecf9  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal,UID:159008f3edd5171bb1bb440b936be3a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" in 1.929s (1.929s including waiting). Image size: 282837706 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.069953273 +0000 UTC m=+3.524828441,LastTimestamp:2026-09-10 13:28:39.069953273 +0000 UTC m=+3.524828441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.091805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.091784    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:39.092901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.092883    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:39.092980 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.092915    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:39.092980 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.092926    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:39.126736 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.126633    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-48.ec2.internal.18d3f8b1b43b93bd  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-48.ec2.internal,UID:13f9881a5f465bf70dd944de0203fca5,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\" in 1.972s (1.972s including waiting). Image size: 321671430 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.112922045 +0000 UTC m=+3.567797212,LastTimestamp:2026-09-10 13:28:39.112922045 +0000 UTC m=+3.567797212,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.197147 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.197069    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-48.ec2.internal.18d3f8b1b86641c1  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-48.ec2.internal,UID:13f9881a5f465bf70dd944de0203fca5,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.182827969 +0000 UTC m=+3.637703136,LastTimestamp:2026-09-10 13:28:39.182827969 +0000 UTC m=+3.637703136,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.199932 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.199908    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:39.201186 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.201168    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:39.201275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.201198    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:39.201275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.201209    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:39.201275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.201235    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:39.220212 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.220088    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-133-48.ec2.internal.18d3f8b1b8e2ad30  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-133-48.ec2.internal,UID:13f9881a5f465bf70dd944de0203fca5,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.190981936 +0000 UTC m=+3.645857095,LastTimestamp:2026-09-10 13:28:39.190981936 +0000 UTC m=+3.645857095,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.230909 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.230884    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:39.305136 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.305105    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:28:39.370918 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.370842    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal.18d3f8b1c2fd020f  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal,UID:159008f3edd5171bb1bb440b936be3a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.360479759 +0000 UTC m=+3.815354922,LastTimestamp:2026-09-10 13:28:39.360479759 +0000 UTC m=+3.815354922,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.383684 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.383614    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal.18d3f8b1c37d7011  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal,UID:159008f3edd5171bb1bb440b936be3a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-133-48.ec2.internal,},FirstTimestamp:2026-09-10 13:28:39.368896529 +0000 UTC m=+3.823771688,LastTimestamp:2026-09-10 13:28:39.368896529 +0000 UTC m=+3.823771688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-48.ec2.internal,}"
Sep 10 13:28:39.782141 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.782104    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:28:39.961423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:39.961385    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:39.980646 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:39.980615    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:28:40.071012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.070924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal" event={"ID":"13f9881a5f465bf70dd944de0203fca5","Type":"ContainerStarted","Data":"7292fa822bdbc3e33f8471bdb61d5a94a233bafa7a00c89392a8e005d8123757"}
Sep 10 13:28:40.071012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.070951    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:40.072497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.072476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerStarted","Data":"6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2"}
Sep 10 13:28:40.072605 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.072548    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:40.073151 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073136    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:40.073211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073161    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:40.073211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073170    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:40.073211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073179    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:40.073211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073192    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:40.073211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.073203    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:40.073371 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:40.073349    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:40.073371 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:40.073357    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:40.968745 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:40.968660    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:41.074113 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.074085    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:41.074532 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.074096    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:41.075044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075022    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:41.075044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075032    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:41.075182 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075058    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:41.075182 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075062    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:41.075182 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075073    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:41.075182 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.075079    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:41.075328 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:41.075314    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:41.075444 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:41.075429    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:41.961911 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:41.961882    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:42.206419 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:42.206386    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Sep 10 13:28:42.431886 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.431803    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:42.432887 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.432866    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:42.433006 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.432905    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:42.433006 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.432917    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:42.433006 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.432946    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:42.453966 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:42.453933    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:42.962497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:42.962467    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:43.257138 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:43.257051    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:28:43.771666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:43.771633    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:43.772756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:43.772734    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:43.772882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:43.772767    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:43.772882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:43.772782    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:43.962250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:43.962215    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:44.465952 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:44.465914    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:28:44.744024 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:44.743948    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:28:44.820285 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:44.820250    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:28:44.960938 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:44.960911    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:45.960772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:45.960736    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:46.002299 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:46.002268    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-48.ec2.internal\" not found"
Sep 10 13:28:46.964237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:46.964201    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:47.965580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:47.965542    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:48.615901 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:48.615866    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:28:48.854786 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.854746    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:48.855769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.855749    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:48.855863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.855788    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:48.855863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.855801    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:48.855863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.855829    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:48.870936 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:48.870878    2577 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:48.962798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:48.962772    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:49.960305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:49.960274    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:28:50.653708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:50.653671    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78v7x"
Sep 10 13:28:50.886620 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:50.886529    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 13:28:50.984350 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:50.984321    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.017866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.017840    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.079190 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.079166    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.091675 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.091651    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_159008f3edd5171bb1bb440b936be3a3/setup/0.log"
Sep 10 13:28:51.091783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.091688    2577 generic.go:356] "Generic (PLEG): container finished" podID="159008f3edd5171bb1bb440b936be3a3" containerID="6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2" exitCode=1
Sep 10 13:28:51.091783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.091721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerDied","Data":"6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2"}
Sep 10 13:28:51.091966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.091881    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:51.092770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.092756    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:51.092875 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.092789    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:51.092875 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.092803    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:51.093030 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:51.093016    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:51.344643 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.344613    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.344643 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:51.344640    2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.384225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.384201    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.409070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.409012    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.483022 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.483005    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.606419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.606394    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:28:51.661577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.661478    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:23:50 +0000 UTC" deadline="2028-06-01 08:18:26.155656852 +0000 UTC"
Sep 10 13:28:51.661577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.661542    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15114h49m34.49411808s"
Sep 10 13:28:51.736690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.736669    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:28:51.767324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.767305    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.767324 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:51.767325    2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:51.943248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.943177    2577 apiserver.go:52] "Watching apiserver"
Sep 10 13:28:51.946991 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.946975    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 13:28:51.947077 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.947064    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=[]
Sep 10 13:28:51.954896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:51.954884    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 13:28:52.023636 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.023612    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:52.052768 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.052752    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:52.094751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.094730    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_159008f3edd5171bb1bb440b936be3a3/setup/0.log"
Sep 10 13:28:52.094838 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.094768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerStarted","Data":"1383e8992e6613744285675ab633fcd3f34159bce02a1918109489b0c5ab238c"}
Sep 10 13:28:52.094888 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.094867    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:52.095809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.095791    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:52.095903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.095823    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:52.095903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.095837    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:52.096064 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:52.096051    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:52.114778 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.114756    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:52.244894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.244827    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:28:52.401294 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.401266    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:52.401294 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:52.401293    2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:52.597475 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.597398    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:52.598485 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.598461    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:52.598621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.598494    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:52.598621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:52.598518    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:53.395079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:53.395049    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:28:54.343075 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:54.342888    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:54.362036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:54.362018    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:54.421754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:54.421738    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:54.694860 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:54.694790    2577 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:54.694860 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:54.694810    2577 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-133-48.ec2.internal" not found
Sep 10 13:28:55.624710 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:55.624680    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-48.ec2.internal\" not found" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:55.871010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.870961    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:28:55.872090 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.872073    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:28:55.872217 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.872106    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:28:55.872217 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.872119    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:28:55.872217 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.872151    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:55.880822 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.880760    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-48.ec2.internal"
Sep 10 13:28:55.880822 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:55.880784    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-48.ec2.internal\": node \"ip-10-0-133-48.ec2.internal\" not found"
Sep 10 13:28:55.903865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.903841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-7dzc2"]
Sep 10 13:28:55.906231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.906213    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gxwt2"]
Sep 10 13:28:55.906362 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.906346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:55.908403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.908385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:55.908785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.908765    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 13:28:55.908881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.908797    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 13:28:55.909417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.909401    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 13:28:55.909477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.909407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 13:28:55.909564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.909557    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 13:28:55.909604 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.909580    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-628mh\""
Sep 10 13:28:55.910453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.910440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bpkml\""
Sep 10 13:28:55.910639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.910627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 13:28:55.910767 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.910752    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 13:28:55.911794 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.911778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:28:55.918438 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.918423    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-fzppt"]
Sep 10 13:28:55.920682 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.920667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:55.920757 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:55.920717    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:28:55.921517 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.921491    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sdzbv"]
Sep 10 13:28:55.923715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.923702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:55.925748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.925731    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b2hqk\""
Sep 10 13:28:55.925846 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.925830    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 13:28:55.925900 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.925836    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 13:28:55.939670 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.939650    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nkwgj"]
Sep 10 13:28:55.941887 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.941874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:55.944544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.944527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 13:28:55.944627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.944601    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 13:28:55.944801 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.944782    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 13:28:55.944801 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.944794    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hjbdc"]
Sep 10 13:28:55.944933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.944786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-497bg\""
Sep 10 13:28:55.947171 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.947155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:55.949146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.949130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67hsw\""
Sep 10 13:28:55.949397 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.949386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 13:28:55.954048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.954034    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:55.960834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.960818    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-t4xx4"]
Sep 10 13:28:55.963073 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.963060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:55.963123 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:55.963112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:28:55.963471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.963458    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 13:28:55.966907 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.966893    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-dmqxk"]
Sep 10 13:28:55.969074 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.969061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:55.973093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.973077    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:28:55.973193 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.973078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 13:28:55.973193 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.973123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4xxgx\""
Sep 10 13:28:55.977095 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.977082    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:28:55.977168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.977140    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"
Sep 10 13:28:55.979138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.979115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal"]
Sep 10 13:28:55.987457 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.987443    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:28:55.988320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.988306    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hgrqp"]
Sep 10 13:28:55.990953 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.990940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:55.998726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.998709    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal"]
Sep 10 13:28:55.999561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.999546    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:28:55.999626 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.999550    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 13:28:55.999785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.999769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 13:28:55.999861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:55.999777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kqtrd\""
Sep 10 13:28:56.000158 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.000144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 13:28:56.000210 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.000144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 13:28:56.000501 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.000488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 13:28:56.000572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.000537    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 13:28:56.029279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.029256    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl8pd"
Sep 10 13:28:56.037646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.037626    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gl8pd"
Sep 10 13:28:56.038733 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq7p\" (UniqueName: \"kubernetes.io/projected/f8dea817-8747-48a0-9536-7d5df8dec6d2-kube-api-access-bsq7p\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.038805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2cs\" (UniqueName: \"kubernetes.io/projected/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-kube-api-access-qh2cs\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.038805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:56.038884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-agent-certs\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.038884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-socket-dir-parent\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.038884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-daemon-config\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.038884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-host\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g24v\" (UniqueName: \"kubernetes.io/projected/9160df28-24ae-4948-af5b-3b8b37a617f1-kube-api-access-2g24v\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-kubernetes\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-system-cni-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.038980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-cnibin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-conf-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8dea817-8747-48a0-9536-7d5df8dec6d2-iptables-alerter-script\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-systemd\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-cni-binary-copy\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-bin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dea817-8747-48a0-9536-7d5df8dec6d2-host-slash\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-conf\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-lib-modules\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-netns\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-serviceca\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-sys\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qft6q\" (UniqueName: \"kubernetes.io/projected/d0e17b82-b1f5-491b-93ea-46303a0c05c9-kube-api-access-qft6q\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-etc-kubernetes\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9xq\" (UniqueName: \"kubernetes.io/projected/7567df29-f6d2-49d8-9bc8-b793630a47f1-kube-api-access-bh9xq\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-host\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-etc-tuned\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-tmp\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cnibin\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysconfig\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-run\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-konnectivity-ca\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-system-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-k8s-cni-cncf-io\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-kubelet\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-hostroot\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.039865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-multus-certs\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.040320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npnz\" (UniqueName: \"kubernetes.io/projected/5296d7be-fb04-4916-908a-4321e9bea195-kube-api-access-5npnz\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.040320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-modprobe-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.040320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-var-lib-kubelet\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.040320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.039743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-multus\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.110630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.110583    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-48.ec2.internal" podStartSLOduration=1.110567515 podStartE2EDuration="1.110567515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:56.091633342 +0000 UTC m=+20.546508505" watchObservedRunningTime="2026-09-10 13:28:56.110567515 +0000 UTC m=+20.565442680"
Sep 10 13:28:56.140905 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-etc-kubernetes\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.140905 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.140905 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkx8\" (UniqueName: \"kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.141041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-etc-tuned\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-etc-kubernetes\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.140956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cnibin\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.141041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-system-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cnibin\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-k8s-cni-cncf-io\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-k8s-cni-cncf-io\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-kubelet\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-system-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-kubelet\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-run\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-hostroot\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141172    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-multus-certs\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-hostroot\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-run\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-multus-certs\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npnz\" (UniqueName: \"kubernetes.io/projected/5296d7be-fb04-4916-908a-4321e9bea195-kube-api-access-5npnz\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-modprobe-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-multus\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2cs\" (UniqueName: \"kubernetes.io/projected/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-kube-api-access-qh2cs\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-multus\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-socket-dir-parent\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g24v\" (UniqueName: \"kubernetes.io/projected/9160df28-24ae-4948-af5b-3b8b37a617f1-kube-api-access-2g24v\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-modprobe-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-cnibin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-socket-dir-parent\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-conf-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.141865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-cnibin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-cni-binary-copy\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-bin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-conf-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-var-lib-cni-bin\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-cni-dir\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.141932    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.141973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.141998    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:56.641966495 +0000 UTC m=+21.096841637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-serviceca\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-sys\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qft6q\" (UniqueName: \"kubernetes.io/projected/d0e17b82-b1f5-491b-93ea-46303a0c05c9-kube-api-access-qft6q\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.142688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9xq\" (UniqueName: \"kubernetes.io/projected/7567df29-f6d2-49d8-9bc8-b793630a47f1-kube-api-access-bh9xq\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-host\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-tmp\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-sys\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-host\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-cni-binary-copy\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysconfig\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-konnectivity-ca\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq7p\" (UniqueName: \"kubernetes.io/projected/f8dea817-8747-48a0-9536-7d5df8dec6d2-kube-api-access-bsq7p\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-serviceca\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-var-lib-kubelet\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-var-lib-kubelet\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysconfig\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-agent-certs\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-daemon-config\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0e17b82-b1f5-491b-93ea-46303a0c05c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-host\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-kubernetes\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-host\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-kubernetes\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.143992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-konnectivity-ca\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-system-cni-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.142985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8dea817-8747-48a0-9536-7d5df8dec6d2-iptables-alerter-script\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-systemd\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dea817-8747-48a0-9536-7d5df8dec6d2-host-slash\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-systemd\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0e17b82-b1f5-491b-93ea-46303a0c05c9-system-cni-dir\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8dea817-8747-48a0-9536-7d5df8dec6d2-host-slash\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-conf\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-lib-modules\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7567df29-f6d2-49d8-9bc8-b793630a47f1-multus-daemon-config\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-netns\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-d\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7567df29-f6d2-49d8-9bc8-b793630a47f1-host-run-netns\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.144529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-lib-modules\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.145062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5296d7be-fb04-4916-908a-4321e9bea195-etc-sysctl-conf\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.145062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.143450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8dea817-8747-48a0-9536-7d5df8dec6d2-iptables-alerter-script\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.145668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.145651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-etc-tuned\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.145752 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.145700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5296d7be-fb04-4916-908a-4321e9bea195-tmp\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.146440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.146420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ac85a2f-e430-4f7a-8db3-8b2f6b345143-agent-certs\") pod \"konnectivity-agent-sdzbv\" (UID: \"3ac85a2f-e430-4f7a-8db3-8b2f6b345143\") " pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.167566 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.167548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npnz\" (UniqueName: \"kubernetes.io/projected/5296d7be-fb04-4916-908a-4321e9bea195-kube-api-access-5npnz\") pod \"tuned-dmqxk\" (UID: \"5296d7be-fb04-4916-908a-4321e9bea195\") " pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.169615 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.169602    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:28:56.169672 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.169619    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:28:56.169672 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.169628    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:56.169672 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.169671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:56.669659582 +0000 UTC m=+21.124534755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:56.172658 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.172634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qft6q\" (UniqueName: \"kubernetes.io/projected/d0e17b82-b1f5-491b-93ea-46303a0c05c9-kube-api-access-qft6q\") pod \"multus-additional-cni-plugins-7dzc2\" (UID: \"d0e17b82-b1f5-491b-93ea-46303a0c05c9\") " pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.173247 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.173229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2cs\" (UniqueName: \"kubernetes.io/projected/39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e-kube-api-access-qh2cs\") pod \"node-ca-nkwgj\" (UID: \"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e\") " pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.173463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.173445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq7p\" (UniqueName: \"kubernetes.io/projected/f8dea817-8747-48a0-9536-7d5df8dec6d2-kube-api-access-bsq7p\") pod \"iptables-alerter-gxwt2\" (UID: \"f8dea817-8747-48a0-9536-7d5df8dec6d2\") " pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.173676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.173659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9xq\" (UniqueName: \"kubernetes.io/projected/7567df29-f6d2-49d8-9bc8-b793630a47f1-kube-api-access-bh9xq\") pod \"multus-hjbdc\" (UID: \"7567df29-f6d2-49d8-9bc8-b793630a47f1\") " pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.175595 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.175576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g24v\" (UniqueName: \"kubernetes.io/projected/9160df28-24ae-4948-af5b-3b8b37a617f1-kube-api-access-2g24v\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.217198 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.217175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7dzc2"
Sep 10 13:28:56.221712 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.221693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gxwt2"
Sep 10 13:28:56.230495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.230474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:28:56.244288 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244461 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244461 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.244722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkx8\" (UniqueName: \"kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.244994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.245241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.245567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.245330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.246421 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.246406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.249410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.249395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nkwgj"
Sep 10 13:28:56.253985 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:56.253964    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dea817_8747_48a0_9536_7d5df8dec6d2.slice/crio-39450a557616cdcb7aacf52f86221d272e605c726ca51c63738440e22782eb34 WatchSource:0}: Error finding container 39450a557616cdcb7aacf52f86221d272e605c726ca51c63738440e22782eb34: Status 404 returned error can't find the container with id 39450a557616cdcb7aacf52f86221d272e605c726ca51c63738440e22782eb34
Sep 10 13:28:56.254415 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.254398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjbdc"
Sep 10 13:28:56.254827 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:56.254436    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac85a2f_e430_4f7a_8db3_8b2f6b345143.slice/crio-c736bc7a3462bbdc828acdd078aea1f43047e17a3fca4d0ace0411e775594b77 WatchSource:0}: Error finding container c736bc7a3462bbdc828acdd078aea1f43047e17a3fca4d0ace0411e775594b77: Status 404 returned error can't find the container with id c736bc7a3462bbdc828acdd078aea1f43047e17a3fca4d0ace0411e775594b77
Sep 10 13:28:56.255916 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.255850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkx8\" (UniqueName: \"kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8\") pod \"ovnkube-node-hgrqp\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") " pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.261934 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:56.261912    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c2f8f3_bd46_48e1_b1b5_b7a07d99bf9e.slice/crio-e2bc1c846dcb3ad07b6cb99896114547629617cbd7d84f72247c954ab6b6aa6c WatchSource:0}: Error finding container e2bc1c846dcb3ad07b6cb99896114547629617cbd7d84f72247c954ab6b6aa6c: Status 404 returned error can't find the container with id e2bc1c846dcb3ad07b6cb99896114547629617cbd7d84f72247c954ab6b6aa6c
Sep 10 13:28:56.264824 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:56.264799    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7567df29_f6d2_49d8_9bc8_b793630a47f1.slice/crio-bf19091988c2278544da9ba8329936f8a8d3759901abcbffb20d04c4774b5aad WatchSource:0}: Error finding container bf19091988c2278544da9ba8329936f8a8d3759901abcbffb20d04c4774b5aad: Status 404 returned error can't find the container with id bf19091988c2278544da9ba8329936f8a8d3759901abcbffb20d04c4774b5aad
Sep 10 13:28:56.275676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.275660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmqxk"
Sep 10 13:28:56.283923 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:28:56.283905    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5296d7be_fb04_4916_908a_4321e9bea195.slice/crio-038f0e5109cde1a1ba0291e493cc1a7bd229ecb29a93ef64ecd8fb51129575df WatchSource:0}: Error finding container 038f0e5109cde1a1ba0291e493cc1a7bd229ecb29a93ef64ecd8fb51129575df: Status 404 returned error can't find the container with id 038f0e5109cde1a1ba0291e493cc1a7bd229ecb29a93ef64ecd8fb51129575df
Sep 10 13:28:56.298192 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.298171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:28:56.646876 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.646793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:56.647414 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.646973    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:56.647414 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.647058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:57.647036818 +0000 UTC m=+22.101911966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:56.748085 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:56.747375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:56.748085 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.747587    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:28:56.748085 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.747609    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:28:56.748085 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.747622    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:56.748085 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:56.747679    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:57.747661037 +0000 UTC m=+22.202536197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:57.040563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.040453    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:23:56 +0000 UTC" deadline="2028-06-02 11:25:53.108307092 +0000 UTC"
Sep 10 13:28:57.040563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.040496    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15141h56m56.067816706s"
Sep 10 13:28:57.107255 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.107223    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_159008f3edd5171bb1bb440b936be3a3/setup/0.log"
Sep 10 13:28:57.107422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.107267    2577 generic.go:356] "Generic (PLEG): container finished" podID="159008f3edd5171bb1bb440b936be3a3" containerID="1383e8992e6613744285675ab633fcd3f34159bce02a1918109489b0c5ab238c" exitCode=0
Sep 10 13:28:57.107422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.107343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerDied","Data":"1383e8992e6613744285675ab633fcd3f34159bce02a1918109489b0c5ab238c"}
Sep 10 13:28:57.107422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.107398    2577 scope.go:117] "RemoveContainer" containerID="6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2"
Sep 10 13:28:57.107817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.107798    2577 scope.go:117] "RemoveContainer" containerID="6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2"
Sep 10 13:28:57.111932 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.111908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdzbv" event={"ID":"3ac85a2f-e430-4f7a-8db3-8b2f6b345143","Type":"ContainerStarted","Data":"c736bc7a3462bbdc828acdd078aea1f43047e17a3fca4d0ace0411e775594b77"}
Sep 10 13:28:57.117768 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.117742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gxwt2" event={"ID":"f8dea817-8747-48a0-9536-7d5df8dec6d2","Type":"ContainerStarted","Data":"39450a557616cdcb7aacf52f86221d272e605c726ca51c63738440e22782eb34"}
Sep 10 13:28:57.119889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.119783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"836dae833b5300ede15229989463c5e287e047b613357f85379760b590352a55"}
Sep 10 13:28:57.124175 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.124120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmqxk" event={"ID":"5296d7be-fb04-4916-908a-4321e9bea195","Type":"ContainerStarted","Data":"038f0e5109cde1a1ba0291e493cc1a7bd229ecb29a93ef64ecd8fb51129575df"}
Sep 10 13:28:57.131255 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.131195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerStarted","Data":"bf19091988c2278544da9ba8329936f8a8d3759901abcbffb20d04c4774b5aad"}
Sep 10 13:28:57.136073 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.136000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkwgj" event={"ID":"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e","Type":"ContainerStarted","Data":"e2bc1c846dcb3ad07b6cb99896114547629617cbd7d84f72247c954ab6b6aa6c"}
Sep 10 13:28:57.136870 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.136759    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_openshift-machine-config-operator_159008f3edd5171bb1bb440b936be3a3_0 in pod sandbox a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b from index: no such id: '6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2'" containerID="6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2"
Sep 10 13:28:57.147694 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.136829    2577 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_openshift-machine-config-operator_159008f3edd5171bb1bb440b936be3a3_0 in pod sandbox a195dec7148b21f5d3d5f2a63bc111a878373896eb2e7f8545774d35fe9fe47b from index: no such id: '6dcf04b8855ab22528fdd063815d97a8593aa267e4e99da23e9397840ae49cd2'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal_openshift-machine-config-operator(159008f3edd5171bb1bb440b936be3a3)\"" logger="UnhandledError"
Sep 10 13:28:57.155669 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.155633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerStarted","Data":"685d334b91bcb522ef6788798396591bf003ae056eb9b04e80e1a8eef6cdaf47"}
Sep 10 13:28:57.656660 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.656574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:57.657242 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.656756    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:57.657242 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.656830    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:59.65680218 +0000 UTC m=+24.111677336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:57.757000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:57.756957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:57.757192 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.757144    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:28:57.757251 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.757192    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:28:57.757251 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.757208    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:57.757359 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:57.757267    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:28:59.757248575 +0000 UTC m=+24.212123723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:58.041223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:58.041126    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:23:56 +0000 UTC" deadline="2028-04-18 10:01:00.228808683 +0000 UTC"
Sep 10 13:28:58.041223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:58.041172    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14060h32m2.187641466s"
Sep 10 13:28:58.059915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:58.059887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:58.060114 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:58.060017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:28:58.060594 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:58.060566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:58.060691 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:58.060665    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:28:58.162942 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:58.162905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" event={"ID":"159008f3edd5171bb1bb440b936be3a3","Type":"ContainerStarted","Data":"e6d7b53b52fc0d83779749b9ed9e9a6a134cc6c272a0224d4f8032a6d640dea5"}
Sep 10 13:28:59.672592 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:59.672553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:28:59.673185 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.672728    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:59.673185 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.672798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:03.672777886 +0000 UTC m=+28.127653050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:28:59.773471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:28:59.773363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:28:59.773674 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.773571    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:28:59.773674 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.773600    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:28:59.773674 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.773614    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:28:59.773674 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:28:59.773675    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:03.773657606 +0000 UTC m=+28.228532753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:00.061447 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:00.061372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:00.061624 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:00.061466    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:00.061748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:00.061733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:00.061817 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:00.061803    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:00.584906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:00.584846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-48.ec2.internal" podStartSLOduration=5.584822337 podStartE2EDuration="5.584822337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:58.180974777 +0000 UTC m=+22.635849941" watchObservedRunningTime="2026-09-10 13:29:00.584822337 +0000 UTC m=+25.039697499"
Sep 10 13:29:00.585243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:00.585225    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hgrqp"]
Sep 10 13:29:02.058904 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:02.058599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:02.058904 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:02.058722    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:02.058904 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:02.058777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:02.058904 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:02.058866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:03.706686 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:03.706645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:03.707084 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.706788    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:03.707084 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.706853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:11.706835639 +0000 UTC m=+36.161710785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:03.807119 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:03.807079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:03.807293 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.807268    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:29:03.807293 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.807287    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:29:03.807418 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.807299    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:03.807418 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:03.807358    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:11.80734 +0000 UTC m=+36.262215162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:04.057711 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:04.057632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:04.057855 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:04.057768    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:04.058190 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:04.058171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:04.058306 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:04.058285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:06.058416 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:06.058371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:06.058883 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:06.058484    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:06.058883 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:06.058843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:06.059010 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:06.058939    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:08.057825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:08.057786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:08.058331 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:08.057936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:08.058331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:08.057985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:08.058331 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:08.058115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:10.057421 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:10.057323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:10.057947 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:10.057342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:10.057947 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:10.057519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:10.057947 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:10.057621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:11.760065 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:11.760025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:11.760484 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.760158    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:11.760484 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.760219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:27.760201965 +0000 UTC m=+52.215077107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:11.860627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:11.860591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:11.860791 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.860722    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:29:11.860791 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.860738    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:29:11.860791 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.860747    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:11.860896 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:11.860797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:27.86078054 +0000 UTC m=+52.315655683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:12.057478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:12.057407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:12.057631 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:12.057524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:12.057631 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:12.057586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:12.057705 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:12.057680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:13.195183 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.194814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdzbv" event={"ID":"3ac85a2f-e430-4f7a-8db3-8b2f6b345143","Type":"ContainerStarted","Data":"2148d85baa3302e630f95710200b22c09b4c10593e0d525d638e8ba9960d85df"}
Sep 10 13:29:13.197925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.197899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f"}
Sep 10 13:29:13.198107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.197971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"a81e8856e6c3ba5a027d61ed43b655f1c876ef456ff6a851f180767f9c56075c"}
Sep 10 13:29:13.198107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.198013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"f49896dd033690519683cc794fc1eed96b1bc3a4f8c5383d060cd6a6a93cd0a6"}
Sep 10 13:29:13.198107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.198043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"5bf9c428b5a512f68e6aa4b40c9a55067b23650b3f4b5d515e37279510f6d97f"}
Sep 10 13:29:13.198107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.198071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"de8b36c8b8be0bbd31f87430f76064292ef9269777f5e7d2bec7705836477699"}
Sep 10 13:29:13.198107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.198097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"3f6e93a522a8cd96b17b551038175c0fa17329388d3ac666585131e040104d1b"}
Sep 10 13:29:13.199356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.199329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmqxk" event={"ID":"5296d7be-fb04-4916-908a-4321e9bea195","Type":"ContainerStarted","Data":"afbef614a16e81f1da9c37ddaf809f59b58968c9c53f6aad45e5fad6255cc7ca"}
Sep 10 13:29:13.200816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.200785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerStarted","Data":"5e12fa79f6f240eced3fe7e74e9b7296f2fe744a269d2773a0ad74ec7f28bcb6"}
Sep 10 13:29:13.203942 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.203886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nkwgj" event={"ID":"39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e","Type":"ContainerStarted","Data":"add4d6173f3f5c8ee9edea4df6a4eca0cd262b56eb9f7fd6f77ccdbbb7b52d99"}
Sep 10 13:29:13.205394 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.205375    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="d0b1aa6251c0855210e67f5f57390c7af5ef1a1dfc11933831b4eb699764decc" exitCode=0
Sep 10 13:29:13.205488 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.205401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"d0b1aa6251c0855210e67f5f57390c7af5ef1a1dfc11933831b4eb699764decc"}
Sep 10 13:29:13.208399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.208364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sdzbv" podStartSLOduration=2.099983339 podStartE2EDuration="18.208352669s" totalImagesPullingTime="16.10836933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.208122541 +0000 UTC m=+37.662997704" watchObservedRunningTime="2026-09-10 13:29:13.208352669 +0000 UTC m=+37.663227832"
Sep 10 13:29:13.223099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.223064    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nkwgj" podStartSLOduration=2.066554288 podStartE2EDuration="18.223053043s" totalImagesPullingTime="16.156498755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.223005919 +0000 UTC m=+37.677881083" watchObservedRunningTime="2026-09-10 13:29:13.223053043 +0000 UTC m=+37.677928242"
Sep 10 13:29:13.236882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.236843    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hjbdc" podStartSLOduration=2.042767725 podStartE2EDuration="18.236831122s" totalImagesPullingTime="16.194063397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.236581683 +0000 UTC m=+37.691456846" watchObservedRunningTime="2026-09-10 13:29:13.236831122 +0000 UTC m=+37.691706286"
Sep 10 13:29:13.250360 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:13.250322    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dmqxk" podStartSLOduration=2.113573501 podStartE2EDuration="18.25031222s" totalImagesPullingTime="16.136738719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:13.250229017 +0000 UTC m=+37.705104180" watchObservedRunningTime="2026-09-10 13:29:13.25031222 +0000 UTC m=+37.705187382"
Sep 10 13:29:14.057324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:14.057293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:14.057497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:14.057296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:14.057497 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:14.057396    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:14.057497 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:14.057482    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:14.209048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:14.209005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gxwt2" event={"ID":"f8dea817-8747-48a0-9536-7d5df8dec6d2","Type":"ContainerStarted","Data":"5d7600429b73988b7b5421101f80d6ad896eb28397e90421f7f75609438d9d9a"}
Sep 10 13:29:15.214012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:15.213980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888"}
Sep 10 13:29:16.058035 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:16.058004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:16.058189 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:16.058097    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:16.058189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:16.058144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:16.058268 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:16.058198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:16.231500 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:16.231471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:29:16.232177 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:16.232157    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:29:16.247415 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:16.247368    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gxwt2" podStartSLOduration=5.08269747 podStartE2EDuration="21.247349125s" totalImagesPullingTime="16.164651655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:14.227166501 +0000 UTC m=+38.682041666" watchObservedRunningTime="2026-09-10 13:29:16.247349125 +0000 UTC m=+40.702224292"
Sep 10 13:29:18.061032 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.060832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:18.061593 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.060878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:18.061593 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.061121    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:18.061593 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.061234    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:18.231729 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.231689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerStarted","Data":"db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6"}
Sep 10 13:29:18.231934 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.231907    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb" containerID="cri-o://6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" gracePeriod=30
Sep 10 13:29:18.232055 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.231952    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb" containerID="cri-o://fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" gracePeriod=30
Sep 10 13:29:18.232055 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232017    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="northd" containerID="cri-o://a81e8856e6c3ba5a027d61ed43b655f1c876ef456ff6a851f180767f9c56075c" gracePeriod=30
Sep 10 13:29:18.232143 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232099    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://f49896dd033690519683cc794fc1eed96b1bc3a4f8c5383d060cd6a6a93cd0a6" gracePeriod=30
Sep 10 13:29:18.232191 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232171    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-node" containerID="cri-o://5bf9c428b5a512f68e6aa4b40c9a55067b23650b3f4b5d515e37279510f6d97f" gracePeriod=30
Sep 10 13:29:18.232264 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232247    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-acl-logging" containerID="cri-o://de8b36c8b8be0bbd31f87430f76064292ef9269777f5e7d2bec7705836477699" gracePeriod=30
Sep 10 13:29:18.232331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.231901    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-controller" containerID="cri-o://3f6e93a522a8cd96b17b551038175c0fa17329388d3ac666585131e040104d1b" gracePeriod=30
Sep 10 13:29:18.232392 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:29:18.232468 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.232453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:29:18.233744 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.233706    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:18.238593 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.238479    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:18.240131 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.240101    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:18.240216 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:18.240146    2577 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb" probeResult="unknown"
Sep 10 13:29:18.247001 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.246173    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovnkube-controller" containerID="cri-o://db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6" gracePeriod=30
Sep 10 13:29:18.255763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:18.255705    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podStartSLOduration=7.129677598 podStartE2EDuration="23.255684904s" totalImagesPullingTime="16.126007306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:18.254944422 +0000 UTC m=+42.709819623" watchObservedRunningTime="2026-09-10 13:29:18.255684904 +0000 UTC m=+42.710560068"
Sep 10 13:29:19.236619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.236592    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:29:19.237049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237008    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-node/0.log"
Sep 10 13:29:19.237363 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237348    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-acl-logging/0.log"
Sep 10 13:29:19.237778 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237766    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-controller/0.log"
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237797    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" exitCode=0
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237809    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" exitCode=0
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237815    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="a81e8856e6c3ba5a027d61ed43b655f1c876ef456ff6a851f180767f9c56075c" exitCode=0
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237821    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="f49896dd033690519683cc794fc1eed96b1bc3a4f8c5383d060cd6a6a93cd0a6" exitCode=143
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237826    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="5bf9c428b5a512f68e6aa4b40c9a55067b23650b3f4b5d515e37279510f6d97f" exitCode=143
Sep 10 13:29:19.237831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237832    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="de8b36c8b8be0bbd31f87430f76064292ef9269777f5e7d2bec7705836477699" exitCode=143
Sep 10 13:29:19.238000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237837    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="3f6e93a522a8cd96b17b551038175c0fa17329388d3ac666585131e040104d1b" exitCode=143
Sep 10 13:29:19.238000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888"}
Sep 10 13:29:19.238000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f"}
Sep 10 13:29:19.238000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"a81e8856e6c3ba5a027d61ed43b655f1c876ef456ff6a851f180767f9c56075c"}
Sep 10 13:29:19.238000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.237975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"f49896dd033690519683cc794fc1eed96b1bc3a4f8c5383d060cd6a6a93cd0a6"}
Sep 10 13:29:19.238140 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.238002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"5bf9c428b5a512f68e6aa4b40c9a55067b23650b3f4b5d515e37279510f6d97f"}
Sep 10 13:29:19.238140 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.238029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"de8b36c8b8be0bbd31f87430f76064292ef9269777f5e7d2bec7705836477699"}
Sep 10 13:29:19.238140 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:19.238057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"3f6e93a522a8cd96b17b551038175c0fa17329388d3ac666585131e040104d1b"}
Sep 10 13:29:20.058132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.058103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:20.058322 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:20.058208    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:20.058322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.058280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:20.058402 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:20.058375    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:20.099079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.099051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:29:20.099229 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.099136    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 13:29:20.099546 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.099527    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdzbv"
Sep 10 13:29:20.241405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.241373    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="de1893097c9c2c21d9c949e30922a1a66c485af92de963494b0b46d3fbcb2133" exitCode=0
Sep 10 13:29:20.241832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:20.241461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"de1893097c9c2c21d9c949e30922a1a66c485af92de963494b0b46d3fbcb2133"}
Sep 10 13:29:22.057829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:22.057798    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:22.058272 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:22.057886    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:22.058272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:22.057977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:22.058272 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:22.058064    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:22.246876 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:22.246845    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="463bd2a1edd0e1dd8e0b9612b4f3444734e4050b98b71ebf496f622494be57ea" exitCode=0
Sep 10 13:29:22.247054 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:22.246885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"463bd2a1edd0e1dd8e0b9612b4f3444734e4050b98b71ebf496f622494be57ea"}
Sep 10 13:29:24.057761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:24.057724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:24.057761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:24.057764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:24.058175 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:24.057872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:24.058175 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:24.057957    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:25.254422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:25.254383    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="10f7513b528dfeb0e15e1c33a53d89c50e742ed3c48f95168962b846922eb5e4" exitCode=0
Sep 10 13:29:25.254815 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:25.254437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"10f7513b528dfeb0e15e1c33a53d89c50e742ed3c48f95168962b846922eb5e4"}
Sep 10 13:29:26.060124 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:26.060098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:26.060278 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:26.060098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:26.060278 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:26.060194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:26.060353 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:26.060273    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:27.766537 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:27.766486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:27.766957 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.766610    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:27.766957 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.766671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:59.766654275 +0000 UTC m=+84.221529429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:27.867341 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:27.867306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:27.867584 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.867445    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:29:27.867584 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.867461    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:29:27.867584 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.867470    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:27.867584 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:27.867534    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:29:59.867503149 +0000 UTC m=+84.322378290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:28.057308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:28.057229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:28.057501 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:28.057229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:28.057501 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.057363    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:28.057501 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.057418    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:28.232329 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:28.232297    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:29:28.232750 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.232715    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:28.232874 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.232738    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:28.233122 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233086    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:28.233240 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233113    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:28.233530 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233478    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:28.233530 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233479    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:28.233661 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233542    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb" probeResult="unknown"
Sep 10 13:29:28.233661 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:28.233590    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb" probeResult="unknown"
Sep 10 13:29:30.060440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:30.060410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:30.060993 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:30.060448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:30.060993 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:30.060551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:30.060993 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:30.060666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:31.271427 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:31.271390    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="325a13798a01951cf11653815b1a8943ed9534df34ceef6b81fd8243a0914250" exitCode=0
Sep 10 13:29:31.271910 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:31.271437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"325a13798a01951cf11653815b1a8943ed9534df34ceef6b81fd8243a0914250"}
Sep 10 13:29:32.059802 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:32.059769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:32.059968 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:32.059891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:32.059968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:32.059935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:32.060099 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:32.060014    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:32.276705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:32.276673    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e17b82-b1f5-491b-93ea-46303a0c05c9" containerID="7be3e908fa23deeead41b7bb6d39eb733b9fe4673b619dc7dc0c626ed7453f23" exitCode=0
Sep 10 13:29:32.277063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:32.276728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerDied","Data":"7be3e908fa23deeead41b7bb6d39eb733b9fe4673b619dc7dc0c626ed7453f23"}
Sep 10 13:29:33.282220 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:33.282177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" event={"ID":"d0e17b82-b1f5-491b-93ea-46303a0c05c9","Type":"ContainerStarted","Data":"d474423025ccc9b0c2e6339580bd6845a936c22f5c1c577cfbac8aa85f51be62"}
Sep 10 13:29:33.300417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:33.300371    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7dzc2" podStartSLOduration=6.826690494 podStartE2EDuration="38.300356457s" totalImagesPullingTime="31.473665963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:33.300332883 +0000 UTC m=+57.755208045" watchObservedRunningTime="2026-09-10 13:29:33.300356457 +0000 UTC m=+57.755231603"
Sep 10 13:29:34.058038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:34.058008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:34.058209 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:34.058102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:34.058209 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:34.058156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:34.058279 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:34.058244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:36.058272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:36.058232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:36.058272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:36.058271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:36.058843 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:36.058361    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:36.058843 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:36.058430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:38.057401 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:38.057365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:38.057834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:38.057365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:38.057834 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.057478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:38.057834 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.057597    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:38.232822 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.232771    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:38.232981 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.232771    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:38.233105 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233077    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:38.233158 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233137    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:38.233317 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233298    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:38.233362 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233330    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb" probeResult="unknown"
Sep 10 13:29:38.233408 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233364    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:38.233446 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:38.233410    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb" probeResult="unknown"
Sep 10 13:29:40.057816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:40.057788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:40.058175 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:40.057928    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:40.058175 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:40.057973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:40.058175 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:40.058046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:42.057724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:42.057694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:42.058074 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:42.057697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:42.058074 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:42.057794    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:42.058074 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:42.057886    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:44.058017 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:44.057978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:44.058501 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:44.057978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:44.058501 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:44.058086    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:44.058501 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:44.058166    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:46.058230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:46.058194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:46.058601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:46.058273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:46.058601 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:46.058287    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:46.058601 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:46.058347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:48.057387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.057357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:48.057852 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.057487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:48.057852 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.057551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:48.057852 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.057642    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:48.232615 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.232572    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:48.232778 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.232572    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:48.232944 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.232915    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:48.232944 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.232933    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:48.233240 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.233218    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:29:48.233240 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.233231    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"nb\"\n"]
Sep 10 13:29:48.233333 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.233252    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of 6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888 is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb" probeResult="unknown"
Sep 10 13:29:48.233381 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.233259    2577 prober.go:104] "Probe errored" err="rpc error: code = NotFound desc = container is not created or running: checking if PID of fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f is running failed: container process not found" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb" probeResult="unknown"
Sep 10 13:29:48.233663 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.233630    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"]
Sep 10 13:29:48.234454 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.234430    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"]
Sep 10 13:29:48.235212 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.235190    2577 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"]
Sep 10 13:29:48.235276 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:48.235221    2577 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovnkube-controller" probeResult="unknown"
Sep 10 13:29:48.315912 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.315895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovnkube-controller/0.log"
Sep 10 13:29:48.317237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.317220    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:29:48.317658 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.317645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-node/0.log"
Sep 10 13:29:48.318015 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318002    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-acl-logging/0.log"
Sep 10 13:29:48.318386 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318375    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-controller/0.log"
Sep 10 13:29:48.318426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318405    2577 generic.go:356] "Generic (PLEG): container finished" podID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerID="db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6" exitCode=137
Sep 10 13:29:48.318456 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6"}
Sep 10 13:29:48.318535 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp" event={"ID":"b44b1261-db37-4fa6-9d6b-60200accbbb1","Type":"ContainerDied","Data":"836dae833b5300ede15229989463c5e287e047b613357f85379760b590352a55"}
Sep 10 13:29:48.318574 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.318556    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="836dae833b5300ede15229989463c5e287e047b613357f85379760b590352a55"
Sep 10 13:29:48.344394 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.344379    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovnkube-controller/0.log"
Sep 10 13:29:48.345734 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.345719    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:29:48.346147 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.346131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/kube-rbac-proxy-node/0.log"
Sep 10 13:29:48.346516 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.346492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-acl-logging/0.log"
Sep 10 13:29:48.346901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.346888    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hgrqp_b44b1261-db37-4fa6-9d6b-60200accbbb1/ovn-controller/0.log"
Sep 10 13:29:48.348182 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.348169    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:29:48.395886 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.395861    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-g8ln9"]
Sep 10 13:29:48.395995 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.395984    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-node"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.395996    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-node"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396004    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="northd"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396010    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="northd"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396017    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396023    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396029    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:29:48.396033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396042    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396048    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396053    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-acl-logging"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396058    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-acl-logging"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396063    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396072    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovnkube-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396077    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovnkube-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-acl-logging"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396107    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="northd"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="nbdb"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396117    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovn-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396121    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="sbdb"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396125    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="ovnkube-controller"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396130    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-node"
Sep 10 13:29:48.396224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.396134    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:29:48.397619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.397604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.407674 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.407740 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407704    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.407740 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash" (OuterVolumeSpecName: "host-slash") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407767    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.407809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407828    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfkx8\" (UniqueName: \"kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407874    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket" (OuterVolumeSpecName: "log-socket") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log" (OuterVolumeSpecName: "node-log") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407916    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407963    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.407987    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408000    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408037    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408086    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd\") pod \"b44b1261-db37-4fa6-9d6b-60200accbbb1\" (UID: \"b44b1261-db37-4fa6-9d6b-60200accbbb1\") "
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408233    2577 reconciler_common.go:299] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-var-lib-openvswitch\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408246    2577 reconciler_common.go:299] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-systemd-units\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408258    2577 reconciler_common.go:299] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-openvswitch\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408269    2577 reconciler_common.go:299] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-log-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408320    2577 reconciler_common.go:299] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-env-overrides\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408335    2577 reconciler_common.go:299] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-node-log\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408348    2577 reconciler_common.go:299] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-bin\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408360    2577 reconciler_common.go:299] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-cni-netd\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.408842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408373    2577 reconciler_common.go:299] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-netns\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408385    2577 reconciler_common.go:299] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-ovn\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408396    2577 reconciler_common.go:299] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-var-lib-cni-networks-ovn-kubernetes\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408409    2577 reconciler_common.go:299] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-kubelet\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408418    2577 reconciler_common.go:299] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-etc-openvswitch\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408426    2577 reconciler_common.go:299] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-slash\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408434    2577 reconciler_common.go:299] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-script-lib\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.409368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.408441    2577 reconciler_common.go:299] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-host-run-ovn-kubernetes\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.411672 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.411649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:29:48.411672 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.411658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8" (OuterVolumeSpecName: "kube-api-access-sfkx8") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "kube-api-access-sfkx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:29:48.412217 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.412197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "b44b1261-db37-4fa6-9d6b-60200accbbb1" (UID: "b44b1261-db37-4fa6-9d6b-60200accbbb1"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:29:48.509615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-kubelet\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-netns\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-bin\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-env-overrides\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-script-lib\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tvs\" (UniqueName: \"kubernetes.io/projected/602b6dbd-a766-4674-bc7c-5c49b425b045-kube-api-access-r2tvs\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-etc-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-log-socket\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.509803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-config\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-node-log\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-systemd-units\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-systemd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/602b6dbd-a766-4674-bc7c-5c49b425b045-ovn-node-metrics-cert\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-slash\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-var-lib-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-netd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.509989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-ovn\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.510014    2577 reconciler_common.go:299] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovnkube-config\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.510040    2577 reconciler_common.go:299] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b44b1261-db37-4fa6-9d6b-60200accbbb1-ovn-node-metrics-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.510077    2577 reconciler_common.go:299] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b44b1261-db37-4fa6-9d6b-60200accbbb1-run-systemd\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.510108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.510097    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfkx8\" (UniqueName: \"kubernetes.io/projected/b44b1261-db37-4fa6-9d6b-60200accbbb1-kube-api-access-sfkx8\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:29:48.610896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-kubelet\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.610896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-netns\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.610896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-bin\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.610896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-env-overrides\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.610896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-script-lib\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tvs\" (UniqueName: \"kubernetes.io/projected/602b6dbd-a766-4674-bc7c-5c49b425b045-kube-api-access-r2tvs\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-etc-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-netns\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-log-socket\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-bin\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-kubelet\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.610993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-etc-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-log-socket\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-run-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-config\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-node-log\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-node-log\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-systemd-units\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-systemd-units\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-systemd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/602b6dbd-a766-4674-bc7c-5c49b425b045-ovn-node-metrics-cert\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-slash\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-var-lib-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-netd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-ovn\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-var-lib-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-systemd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-openvswitch\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-run-ovn\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-cni-netd\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/602b6dbd-a766-4674-bc7c-5c49b425b045-host-slash\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-env-overrides\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-config\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.611873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.611664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/602b6dbd-a766-4674-bc7c-5c49b425b045-ovnkube-script-lib\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.613380 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.613365    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/602b6dbd-a766-4674-bc7c-5c49b425b045-ovn-node-metrics-cert\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.619721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.619698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tvs\" (UniqueName: \"kubernetes.io/projected/602b6dbd-a766-4674-bc7c-5c49b425b045-kube-api-access-r2tvs\") pod \"ovnkube-node-g8ln9\" (UID: \"602b6dbd-a766-4674-bc7c-5c49b425b045\") " pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.705306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:48.705278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:48.713627 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:29:48.713602    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602b6dbd_a766_4674_bc7c_5c49b425b045.slice/crio-0b8319a7987234e3479d8ba7d361fb8ae4cae7285b094a310e40c2e84e7d3af3 WatchSource:0}: Error finding container 0b8319a7987234e3479d8ba7d361fb8ae4cae7285b094a310e40c2e84e7d3af3: Status 404 returned error can't find the container with id 0b8319a7987234e3479d8ba7d361fb8ae4cae7285b094a310e40c2e84e7d3af3
Sep 10 13:29:49.323680 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"8ea0c500fa70d02797848a3a879926cf9d0bcb121e945485eae1a8978113c089"}
Sep 10 13:29:49.323680 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hgrqp"
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"eeddce5b45cce2a52bb4c919c66192dedde84a066d7ddd295fc6e39695ecd623"}
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"87df4dcbef49bdc1b8b877b98be57a5f7a72f186774e9192a31d9f1372bd2d53"}
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"b5632d58f26dec57ed9c8ebbc76b227ba25573e474650658c86a3c0db1097edb"}
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"efb3c920a376ec00377e37a96c4dfc00c25d1ee7feac03e108d0680df321fde4"}
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"f617aef04a884ca7f04602ed0271da15fcf3bc5ebb007d8a24cbd37af8fba37c"}
Sep 10 13:29:49.324102 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.323893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"0b8319a7987234e3479d8ba7d361fb8ae4cae7285b094a310e40c2e84e7d3af3"}
Sep 10 13:29:49.349641 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.349618    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hgrqp"]
Sep 10 13:29:49.354844 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:49.354824    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hgrqp"]
Sep 10 13:29:50.057409 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:50.057376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:50.057615 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:50.057488    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:50.057615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:50.057569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:50.057742 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:50.057654    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:50.060129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:50.060098    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44b1261-db37-4fa6-9d6b-60200accbbb1" path="/var/lib/kubelet/pods/b44b1261-db37-4fa6-9d6b-60200accbbb1/volumes"
Sep 10 13:29:51.330866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:51.330783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"b12fbe1e1e8b10f617de07286ef39d8fb6ab5fc669478abb506bce0d71382d76"}
Sep 10 13:29:52.059908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:52.059878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:52.060137 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:52.059983    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:52.060137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:52.060053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:52.060137 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:52.060129    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:53.339105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:53.338898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" event={"ID":"602b6dbd-a766-4674-bc7c-5c49b425b045","Type":"ContainerStarted","Data":"b72f9a477cde86bb0d01cf86bb06e195e2cd99a1749ddbe1d1c18a353af03394"}
Sep 10 13:29:53.339554 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:53.339174    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:53.339554 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:53.339217    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:53.355111 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:53.355087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:53.364898 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:53.364859    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" podStartSLOduration=5.36484609 podStartE2EDuration="5.36484609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:53.364570316 +0000 UTC m=+77.819445479" watchObservedRunningTime="2026-09-10 13:29:53.36484609 +0000 UTC m=+77.819721252"
Sep 10 13:29:54.060724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:54.060697    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:54.060896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:54.060703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:54.060896 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:54.060880    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:54.060968 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:54.060792    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:54.341544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:54.341449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:54.355411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:54.355380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:29:56.058005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:56.057977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:56.058354 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:56.058051    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:56.058354 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:56.058094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:56.058354 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:56.058146    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:57.192670 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.192640    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wl48r"]
Sep 10 13:29:57.194201 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.194185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.196395 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.196372    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 13:29:57.196395 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.196379    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 13:29:57.196564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.196449    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glzgh\""
Sep 10 13:29:57.277234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.277199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317a825d-39ad-4035-bffe-ac32f2be5000-hosts-file\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.277234 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.277234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317a825d-39ad-4035-bffe-ac32f2be5000-tmp-dir\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.277480 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.277278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdj5p\" (UniqueName: \"kubernetes.io/projected/317a825d-39ad-4035-bffe-ac32f2be5000-kube-api-access-cdj5p\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.377557 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.377523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317a825d-39ad-4035-bffe-ac32f2be5000-hosts-file\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.377557 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.377558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317a825d-39ad-4035-bffe-ac32f2be5000-tmp-dir\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.377795 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.377588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdj5p\" (UniqueName: \"kubernetes.io/projected/317a825d-39ad-4035-bffe-ac32f2be5000-kube-api-access-cdj5p\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.377795 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.377587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/317a825d-39ad-4035-bffe-ac32f2be5000-hosts-file\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.377894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.377864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/317a825d-39ad-4035-bffe-ac32f2be5000-tmp-dir\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.386164 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.386135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdj5p\" (UniqueName: \"kubernetes.io/projected/317a825d-39ad-4035-bffe-ac32f2be5000-kube-api-access-cdj5p\") pod \"node-resolver-wl48r\" (UID: \"317a825d-39ad-4035-bffe-ac32f2be5000\") " pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.502813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:57.502729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wl48r"
Sep 10 13:29:57.511798 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:29:57.511770    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317a825d_39ad_4035_bffe_ac32f2be5000.slice/crio-6973413a7d5caaa6f6ccbe96cf6d179a578060543d7430d73fd1093a924e9e70 WatchSource:0}: Error finding container 6973413a7d5caaa6f6ccbe96cf6d179a578060543d7430d73fd1093a924e9e70: Status 404 returned error can't find the container with id 6973413a7d5caaa6f6ccbe96cf6d179a578060543d7430d73fd1093a924e9e70
Sep 10 13:29:58.057981 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.057950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:58.058125 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.057962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:58.058125 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:58.058055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:29:58.058214 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:58.058128    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:29:58.351416 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.351375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wl48r" event={"ID":"317a825d-39ad-4035-bffe-ac32f2be5000","Type":"ContainerStarted","Data":"de2c1cd079e81dc080d700becd4619a13024b66a7720fe9318a7c151fcd3ec68"}
Sep 10 13:29:58.351992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.351453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wl48r" event={"ID":"317a825d-39ad-4035-bffe-ac32f2be5000","Type":"ContainerStarted","Data":"6973413a7d5caaa6f6ccbe96cf6d179a578060543d7430d73fd1093a924e9e70"}
Sep 10 13:29:58.352878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.352861    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/0.log"
Sep 10 13:29:58.352932 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.352896    2577 generic.go:356] "Generic (PLEG): container finished" podID="7567df29-f6d2-49d8-9bc8-b793630a47f1" containerID="5e12fa79f6f240eced3fe7e74e9b7296f2fe744a269d2773a0ad74ec7f28bcb6" exitCode=1
Sep 10 13:29:58.352981 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.352929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerDied","Data":"5e12fa79f6f240eced3fe7e74e9b7296f2fe744a269d2773a0ad74ec7f28bcb6"}
Sep 10 13:29:58.353205 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.353194    2577 scope.go:117] "RemoveContainer" containerID="5e12fa79f6f240eced3fe7e74e9b7296f2fe744a269d2773a0ad74ec7f28bcb6"
Sep 10 13:29:58.365748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:58.365711    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wl48r" podStartSLOduration=1.365698887 podStartE2EDuration="1.365698887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:58.365544806 +0000 UTC m=+82.820419970" watchObservedRunningTime="2026-09-10 13:29:58.365698887 +0000 UTC m=+82.820574050"
Sep 10 13:29:59.357004 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:59.356978    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/0.log"
Sep 10 13:29:59.357466 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:59.357038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerStarted","Data":"84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa"}
Sep 10 13:29:59.790772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:59.790736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:29:59.790924 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.790840    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:59.790924 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.790893    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.790879447 +0000 UTC m=+148.245754588 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:29:59.891961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:29:59.891925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:29:59.892107 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.892083    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:29:59.892107 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.892103    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:29:59.892202 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.892113    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:29:59.892202 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:29:59.892165    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.892152413 +0000 UTC m=+148.347027554 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:30:00.057321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:00.057232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:00.057470 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:00.057232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:00.057470 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:00.057350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:00.057470 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:00.057436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:02.057779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:02.057746    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:02.058229 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:02.057789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:02.058229 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:02.057853    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:02.058229 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:02.057909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:04.057521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:04.057477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:04.057964 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:04.057593    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:04.057964 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:04.057681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:04.057964 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:04.057816    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:06.057950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:06.057922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:06.058321 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:06.058015    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:06.058321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:06.058092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:06.058321 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:06.058185    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:08.058274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:08.058232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:08.058274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:08.058232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:08.058837 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:08.058524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:08.058837 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:08.058622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:10.057879 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:10.057852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:10.058370 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:10.057947    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:10.058370 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:10.058015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:10.058370 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:10.058103    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:12.058126 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:12.058091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:12.058526 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:12.058195    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:12.058526 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:12.058249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:12.058526 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:12.058366    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:14.060382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:14.060344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:14.060937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:14.060348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:14.060937 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:14.060436    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:14.060937 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:14.060570    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:16.060316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:16.060287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:16.060786 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:16.060287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:16.060786 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:16.060411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:16.060786 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:16.060468    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:18.058058 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:18.058019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:18.058497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:18.058032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:18.058497 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:18.058238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:18.058497 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:18.058121    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:20.057228 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:20.057195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:20.057628 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:20.057294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:20.057628 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:20.057373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:20.057628 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:20.057459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:22.057886 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:22.057851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:22.058263 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:22.057851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:22.058263 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:22.057984    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:22.058263 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:22.058022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:24.057472 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:24.057441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:24.058003 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:24.057593    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:24.058003 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:24.057885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:24.058003 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:24.057964    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:26.058357 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:26.058325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:26.058738 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:26.058375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:26.058738 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:26.058443    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:26.058738 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:26.058524    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:26.354892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:26.354806    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" podUID="602b6dbd-a766-4674-bc7c-5c49b425b045" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 13:30:28.058123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:28.058093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:28.058551 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:28.058096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:28.058551 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:28.058197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:28.058551 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:28.058283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:30.057204 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:30.057165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:30.057689 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:30.057286    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:30.057689 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:30.057315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:30.057689 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:30.057380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:32.057173 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:32.057138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:32.057571 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:32.057281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:32.057571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:32.057327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:32.057571 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:32.057411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:34.057628 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:34.057587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:34.058138 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:34.057758    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:34.058138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:34.057776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:34.058138 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:34.057878    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:36.017062 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:36.016869    2577 kubelet_node_status.go:448] "Node not becoming ready in time after startup"
Sep 10 13:30:36.017869 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:36.017838    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:30:36.058142 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:36.058115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:36.058278 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:36.058219    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:36.058358 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:36.058341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:36.058473 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:36.058454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:38.057462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:38.057431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:38.057856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:38.057441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:38.057856 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:38.057559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:38.057856 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:38.057622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:40.057293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:40.057254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:40.057684 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:40.057386    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:40.057684 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:40.057428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:40.057684 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:40.057530    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:41.018783 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:41.018746    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:30:42.057792 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:42.057754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:42.058235 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:42.057755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:42.058235 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:42.057869    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:42.058235 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:42.057977    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:44.057737 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.057655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:44.058086 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:44.057775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:44.058086 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.057833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:44.058086 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:44.057939    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:44.456473 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.456445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/1.log"
Sep 10 13:30:44.456880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.456863    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/0.log"
Sep 10 13:30:44.456990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.456900    2577 generic.go:356] "Generic (PLEG): container finished" podID="7567df29-f6d2-49d8-9bc8-b793630a47f1" containerID="84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa" exitCode=1
Sep 10 13:30:44.456990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.456945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerDied","Data":"84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa"}
Sep 10 13:30:44.457093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.457001    2577 scope.go:117] "RemoveContainer" containerID="5e12fa79f6f240eced3fe7e74e9b7296f2fe744a269d2773a0ad74ec7f28bcb6"
Sep 10 13:30:44.457363 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:44.457343    2577 scope.go:117] "RemoveContainer" containerID="84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa"
Sep 10 13:30:44.457544 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:44.457499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 10s restarting failed container=kube-multus pod=multus-hjbdc_openshift-multus(7567df29-f6d2-49d8-9bc8-b793630a47f1)\"" pod="openshift-multus/multus-hjbdc" podUID="7567df29-f6d2-49d8-9bc8-b793630a47f1"
Sep 10 13:30:45.460644 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:45.460615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/1.log"
Sep 10 13:30:46.019227 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:46.019194    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:30:46.058053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:46.058024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:46.058192 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:46.058108    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:46.058235 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:46.058195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:46.058336 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:46.058315    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:48.057834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:48.057790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:48.058287 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:48.057942    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:48.058287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:48.057984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:48.058287 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:48.058069    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:50.057663 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:50.057625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:50.058053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:50.057741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:50.058053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:50.057794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:50.058053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:50.057881    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:51.020360 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:51.020317    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:30:52.057307 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:52.057267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:52.057789 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:52.057271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:52.057789 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:52.057410    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:52.057789 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:52.057455    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:54.057311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:54.057272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:54.057785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:54.057272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:54.057785 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:54.057413    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:54.057785 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:54.057454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:56.021098 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.021059    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:30:56.058146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.058115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:56.058305 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.058228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:56.058305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.058269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:56.058418 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.058349    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:56.154358 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.154330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9slnm"]
Sep 10 13:30:56.158536 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.158520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.158614 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.158585    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:30:56.254717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.254681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-dbus\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.254717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.254717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.254923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.254737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-kubelet-config\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.354298 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.354231    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" podUID="602b6dbd-a766-4674-bc7c-5c49b425b045" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 13:30:56.355462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.355445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-dbus\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.355529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.355471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.355529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.355495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-kubelet-config\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.355610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.355597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-kubelet-config\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.355644 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.355624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13a758ab-ae02-4484-b6f1-0773f7e2a50a-dbus\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.355644 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.355633    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:56.355707 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.355690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:30:56.855675942 +0000 UTC m=+141.310551095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:56.858888 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:56.858837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:56.859059 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.858982    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:56.859059 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:56.859046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:30:57.85903196 +0000 UTC m=+142.313907106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:57.867783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:57.867747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:57.868169 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:57.867889    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:57.868169 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:57.867939    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:30:59.867926864 +0000 UTC m=+144.322802005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:58.057134 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.057103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:58.057259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.057138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:30:58.057259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.057103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:30:58.057259 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:58.057206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:30:58.057416 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:58.057383    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:30:58.057489 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:58.057467    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:30:58.057553 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.057495    2577 scope.go:117] "RemoveContainer" containerID="84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa"
Sep 10 13:30:58.492685 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.492661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/1.log"
Sep 10 13:30:58.492850 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:58.492725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerStarted","Data":"66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6"}
Sep 10 13:30:59.881405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:30:59.881359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:30:59.881930 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:59.881546    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:30:59.881930 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:30:59.881631    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:03.88160966 +0000 UTC m=+148.336484817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:00.057880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:00.057844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:00.057880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:00.057865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:00.058127 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:00.057948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:00.058127 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:00.058008    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:00.058127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:00.058053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:00.058282 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:00.058151    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:01.023959 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:01.022476    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:02.057500 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:02.057448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:02.057500 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:02.057466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:02.058058 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:02.057563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:02.058058 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:02.057583    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:02.058058 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:02.057654    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:02.058226 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:02.058111    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:02.066714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:02.066688    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" podUID="602b6dbd-a766-4674-bc7c-5c49b425b045" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 13:31:03.804098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:03.804067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:03.804539 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.804226    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:03.804539 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.804295    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs podName:9160df28-24ae-4948-af5b-3b8b37a617f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:05.804279648 +0000 UTC m=+270.259154794 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs") pod "network-metrics-daemon-t4xx4" (UID: "9160df28-24ae-4948-af5b-3b8b37a617f1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:31:03.905012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:03.904982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:03.905124 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905093    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:03.905177 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905140    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:11.905127204 +0000 UTC m=+156.360002362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:03.905177 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:03.905089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:03.905177 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905172    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:31:03.905287 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905193    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:31:03.905287 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905205    2577 projected.go:196] Error preparing data for projected volume kube-api-access-296zg for pod openshift-network-diagnostics/network-check-target-fzppt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:03.905287 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:03.905248    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg podName:342e788e-fc18-4211-b510-0be9a5751ba1 nodeName:}" failed. No retries permitted until 2026-09-10 13:33:05.905236283 +0000 UTC m=+270.360111423 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "kube-api-access-296zg" (UniqueName: "kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg") pod "network-check-target-fzppt" (UID: "342e788e-fc18-4211-b510-0be9a5751ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:31:04.060864 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:04.060785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:04.061021 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:04.060880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:04.061021 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:04.060891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:04.061021 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:04.060953    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:04.061021 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:04.060966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:04.061161 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:04.061049    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:06.023050 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:06.023006    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:06.060110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:06.060075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:06.060273 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:06.060075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:06.060273 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:06.060082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:06.060396 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:06.060283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:06.060396 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:06.060167    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:06.060396 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:06.060320    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:08.057879 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:08.057840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:08.058253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:08.057840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:08.058253 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:08.057960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:08.058253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:08.057854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:08.058253 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:08.058028    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:08.058253 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:08.058126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:10.057664 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:10.057625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:10.058053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:10.057672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:10.058053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:10.057748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:10.058053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:10.057765    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:10.058053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:10.057846    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:10.058053 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:10.057935    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:11.024599 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:11.024564    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:11.952937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:11.952886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:11.953338 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:11.953037    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:11.953338 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:11.953101    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:27.953085466 +0000 UTC m=+172.407960611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:12.057696 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:12.057667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:12.057791 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:12.057667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:12.057833 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:12.057785    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:12.057894 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:12.057875    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:12.057935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:12.057922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:12.057993 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:12.057979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:14.058070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.058033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:14.058529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.058032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:14.058529 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:14.058156    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:14.058529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.058032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:14.058529 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:14.058231    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:14.058529 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:14.058336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:14.700361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.700330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"]
Sep 10 13:31:14.702148 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.702133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.704968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.704949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 13:31:14.704968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.704962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 13:31:14.705254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.705239    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 13:31:14.705756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.705742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vqf84\""
Sep 10 13:31:14.870399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-socket-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-device-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-sys-fs\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mnv\" (UniqueName: \"kubernetes.io/projected/558e8933-10a7-4e70-b9bc-665ad562265b-kube-api-access-b4mnv\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.870621 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.870571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-registration-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.971858 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-socket-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.971858 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.971858 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-device-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-sys-fs\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mnv\" (UniqueName: \"kubernetes.io/projected/558e8933-10a7-4e70-b9bc-665ad562265b-kube-api-access-b4mnv\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-etc-selinux\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-registration-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.971970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-device-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.972022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-registration-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.972026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-sys-fs\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.972308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.972062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/558e8933-10a7-4e70-b9bc-665ad562265b-socket-dir\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:14.983567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:14.983542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mnv\" (UniqueName: \"kubernetes.io/projected/558e8933-10a7-4e70-b9bc-665ad562265b-kube-api-access-b4mnv\") pod \"aws-ebs-csi-driver-node-nnmg2\" (UID: \"558e8933-10a7-4e70-b9bc-665ad562265b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:15.009923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:15.009902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2"
Sep 10 13:31:15.018407 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:31:15.018383    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558e8933_10a7_4e70_b9bc_665ad562265b.slice/crio-7214b0e05435da35c89353da2287765224ef6c6c3f61377e4c1ab9cb0e8cbe7d WatchSource:0}: Error finding container 7214b0e05435da35c89353da2287765224ef6c6c3f61377e4c1ab9cb0e8cbe7d: Status 404 returned error can't find the container with id 7214b0e05435da35c89353da2287765224ef6c6c3f61377e4c1ab9cb0e8cbe7d
Sep 10 13:31:15.532020 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:15.531988    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2" event={"ID":"558e8933-10a7-4e70-b9bc-665ad562265b","Type":"ContainerStarted","Data":"7214b0e05435da35c89353da2287765224ef6c6c3f61377e4c1ab9cb0e8cbe7d"}
Sep 10 13:31:16.025018 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:16.024978    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:16.058909 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:16.058875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:16.059087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:16.058980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:16.059087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:16.059018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:16.059087 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:16.059022    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:16.059233 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:16.059104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:16.059233 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:16.059189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:17.539620 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:17.539588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2" event={"ID":"558e8933-10a7-4e70-b9bc-665ad562265b","Type":"ContainerStarted","Data":"b7f292e4a30c1d21039bd37b9f80018995a2d97ec193f3852f2a4d079e19d326"}
Sep 10 13:31:18.057713 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:18.057671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:18.057713 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:18.057677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:18.057713 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:18.057677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:18.058009 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:18.057941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:18.058065 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:18.058045    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:18.058122 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:18.058101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:18.875406 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:18.875369    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 13:31:19.102066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:19.101906    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:31:18.875386345Z","UUID":"8d38e206-59cd-4173-b99f-6cdac4d4eb47","Handler":null,"Name":"","Endpoint":""}
Sep 10 13:31:19.103292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:19.103273    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 13:31:19.103423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:19.103304    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 13:31:19.548655 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:19.548615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2" event={"ID":"558e8933-10a7-4e70-b9bc-665ad562265b","Type":"ContainerStarted","Data":"f49e8704ab6de185c807b140f30ae30e5db96371571360e8941ccaeabd9cb49b"}
Sep 10 13:31:20.057418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:20.057242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:20.057829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:20.057240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:20.057829 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:20.057523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:20.057829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:20.057240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:20.057829 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:20.057632    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:20.057829 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:20.057702    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:20.553403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:20.553371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2" event={"ID":"558e8933-10a7-4e70-b9bc-665ad562265b","Type":"ContainerStarted","Data":"3679101ddfa4d42518f697e2cb771deb1d7af47dab393d00b5cbf59b0ffd9521"}
Sep 10 13:31:20.571874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:20.571826    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nnmg2" podStartSLOduration=2.153792648 podStartE2EDuration="6.5718112s" totalImagesPullingTime="4.418018552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:31:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:20.571431818 +0000 UTC m=+165.026306980" watchObservedRunningTime="2026-09-10 13:31:20.5718112 +0000 UTC m=+165.026686404"
Sep 10 13:31:21.026217 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:21.026177    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:22.057613 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:22.057570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:22.058046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:22.057570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:22.058046 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:22.057692    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:22.058046 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:22.057809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:22.058046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:22.057570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:22.058046 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:22.057929    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:24.057961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:24.057919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:24.057961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:24.057919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:24.058371 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:24.058063    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:24.058371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:24.058110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:24.058371 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:24.058189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:24.058371 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:24.058247    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:26.026873 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:26.026831    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:26.058255 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:26.058217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:26.058432 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:26.058327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:26.058432 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:26.058336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:26.058432 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:26.058425    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:26.058619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:26.058471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:26.058619 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:26.058531    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:27.958538 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:27.958488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:27.958905 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:27.958609    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:27.958905 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:27.958657    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:31:59.958645064 +0000 UTC m=+204.413520218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:28.057417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:28.057388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:28.057417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:28.057410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:28.057619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:28.057390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:28.057619 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:28.057544    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:28.057684 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:28.057624    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:28.057717 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:28.057683    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:30.058230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:30.058142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:30.058637 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:30.058155    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:30.058637 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:30.058294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:30.058637 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:30.058367    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:30.058637 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:30.058391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:30.058637 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:30.058485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:31.028235 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:31.028200    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:32.057939 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:32.057896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:32.058417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:32.057908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:32.058417 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:32.058028    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:32.058417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:32.058042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:32.058417 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:32.058122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:32.058417 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:32.058190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:32.067263 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:32.067232    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9" podUID="602b6dbd-a766-4674-bc7c-5c49b425b045" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 13:31:34.057525 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:34.057474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:34.057943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:34.057474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:34.057943 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:34.057613    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:34.057943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:34.057477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:34.057943 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:34.057660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:34.057943 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:34.057787    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:36.028639 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:36.028601    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:36.058071 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:36.058044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:36.058228 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:36.058141    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:36.058228 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:36.058211    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:36.058311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:36.058294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:36.058345 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:36.058307    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:36.058399 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:36.058352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:38.057517 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:38.057469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:38.057941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:38.057471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:38.057941 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:38.057597    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:38.057941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:38.057471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:38.057941 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:38.057680    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:38.057941 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:38.057762    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:40.057803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:40.057765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:40.058303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:40.057864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:40.058303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:40.057962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:40.058303 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:40.057956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:40.058303 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:40.058104    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:40.058303 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:40.058198    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:41.030181 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:41.030142    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:42.057945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:42.057902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:42.058412 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:42.057902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:42.058412 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:42.058036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:42.058412 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:42.057923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:42.058412 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:42.058109    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:42.058412 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:42.058171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:43.623857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.623830    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:31:43.624417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.624259    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/1.log"
Sep 10 13:31:43.624417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.624296    2577 generic.go:356] "Generic (PLEG): container finished" podID="7567df29-f6d2-49d8-9bc8-b793630a47f1" containerID="66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6" exitCode=1
Sep 10 13:31:43.624417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.624331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerDied","Data":"66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6"}
Sep 10 13:31:43.624417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.624386    2577 scope.go:117] "RemoveContainer" containerID="84fed308538f127591156e235fed2f3a478cc831d5584697402c6876a9daebaa"
Sep 10 13:31:43.624727 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:43.624708    2577 scope.go:117] "RemoveContainer" containerID="66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6"
Sep 10 13:31:43.624890 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:43.624872    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-multus pod=multus-hjbdc_openshift-multus(7567df29-f6d2-49d8-9bc8-b793630a47f1)\"" pod="openshift-multus/multus-hjbdc" podUID="7567df29-f6d2-49d8-9bc8-b793630a47f1"
Sep 10 13:31:44.057368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:44.057268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:44.057561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:44.057268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:44.057561 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:44.057418    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:44.057561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:44.057276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:44.057561 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:44.057486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:44.057790 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:44.057559    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:44.628524 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:44.628476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:31:45.181018 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.180809    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9slnm"]
Sep 10 13:31:45.181192 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.181114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:45.181252 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:45.181228    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:45.183568 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.183498    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4xx4"]
Sep 10 13:31:45.183701 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.183614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:45.183755 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:45.183721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:45.184213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.184193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fzppt"]
Sep 10 13:31:45.184316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:45.184265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:45.184378 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:45.184340    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:46.031144 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:46.031057    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:47.058157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:47.058119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:47.058157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:47.058150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:47.058686 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:47.058228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:47.058686 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:47.058253    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:47.058686 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:47.058316    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:47.058686 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:47.058371    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:49.058193 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:49.058147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:49.058599 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:49.058153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:49.058599 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:49.058268    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:49.058599 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:49.058153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:49.058599 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:49.058342    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:49.058599 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:49.058430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:51.031911 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:51.031871    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:51.058180 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:51.058150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:51.058336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:51.058151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:51.058336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:51.058151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:51.058411 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:51.058350    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:51.058494 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:51.058470    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:51.058561 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:51.058545    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:53.057732 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:53.057700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:53.058066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:53.057708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:53.058066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:53.057716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:53.058066 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:53.057904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:53.058066 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:53.057946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:53.058066 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:53.058020    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:55.057450 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:55.057345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:55.057450 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:55.057382    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:55.057973 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:55.057490    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:55.057973 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:55.057567    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:55.057973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:55.057587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:55.057973 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:55.057670    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:55.057973 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:55.057932    2577 scope.go:117] "RemoveContainer" containerID="66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6"
Sep 10 13:31:55.058141 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:55.058060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with CrashLoopBackOff: \"back-off 20s restarting failed container=kube-multus pod=multus-hjbdc_openshift-multus(7567df29-f6d2-49d8-9bc8-b793630a47f1)\"" pod="openshift-multus/multus-hjbdc" podUID="7567df29-f6d2-49d8-9bc8-b793630a47f1"
Sep 10 13:31:56.032347 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:56.032309    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:31:57.058161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:57.058120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:57.058161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:57.058149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:57.058161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:57.058164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:57.058670 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:57.058258    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:57.058670 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:57.058293    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:57.058670 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:57.058377    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:59.057218 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:59.057163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:59.057719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:59.057163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:31:59.057719 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:59.057291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:31:59.057719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:59.057163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:31:59.057719 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:59.057359    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:31:59.057719 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:59.057438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:31:59.963034 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:31:59.962796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:31:59.963239 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:59.962927    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:31:59.963386 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:31:59.963362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret podName:13a758ab-ae02-4484-b6f1-0773f7e2a50a nodeName:}" failed. No retries permitted until 2026-09-10 13:33:03.963339514 +0000 UTC m=+268.418214659 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret") pod "global-pull-secret-syncer-9slnm" (UID: "13a758ab-ae02-4484-b6f1-0773f7e2a50a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:32:01.033756 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:01.033716    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:32:01.058035 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:01.058006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:01.058163 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:01.058048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:01.058163 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:01.058048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:01.058163 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:01.058122    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:32:01.058264 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:01.058224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:32:01.058298 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:01.058285    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:32:02.067645 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:02.067620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g8ln9"
Sep 10 13:32:03.057768 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:03.057717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:03.057957 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:03.057722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:03.057957 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:03.057840    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:32:03.057957 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:03.057916    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:32:03.057957 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:03.057747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:03.058094 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:03.057991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:32:05.057836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:05.057802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:05.058288 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:05.057802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:05.058288 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:05.057926    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:32:05.058288 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:05.057817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:05.058288 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:05.057974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:32:05.058288 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:05.058055    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:32:06.034556 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:06.034503    2577 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:32:06.058165 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:06.058141    2577 scope.go:117] "RemoveContainer" containerID="66edebf4364ee95665f066e3fc6d1c10679c7c460ef8069674620efdb68dfef6"
Sep 10 13:32:06.689670 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:06.689643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:32:06.689835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:06.689708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjbdc" event={"ID":"7567df29-f6d2-49d8-9bc8-b793630a47f1","Type":"ContainerStarted","Data":"a49607c4d40dab9115b73d9169824ccd9cd37a9b6c236a701c4bc6590c044dee"}
Sep 10 13:32:07.057976 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:07.057885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:07.058137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:07.057885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:07.058137 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:07.057997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:32:07.058137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:07.057885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:07.058137 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:07.058077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:32:07.058499 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:07.058151    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:32:09.057714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:09.057677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:09.058123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:09.057678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:09.058123 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:09.057801    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4xx4" podUID="9160df28-24ae-4948-af5b-3b8b37a617f1"
Sep 10 13:32:09.058123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:09.057678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:09.058123 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:09.057883    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fzppt" podUID="342e788e-fc18-4211-b510-0be9a5751ba1"
Sep 10 13:32:09.058123 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:09.057932    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9slnm" podUID="13a758ab-ae02-4484-b6f1-0773f7e2a50a"
Sep 10 13:32:11.058152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.058123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:32:11.058719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.058123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:32:11.058719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.058123    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:32:11.061539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061492    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 13:32:11.061539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\""
Sep 10 13:32:11.061693 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061593    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 13:32:11.061693 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 13:32:11.061865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 13:32:11.061905 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:11.061867    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\""
Sep 10 13:32:14.356015 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:14.355990    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wl48r_317a825d-39ad-4035-bffe-ac32f2be5000/dns-node-resolver/0.log"
Sep 10 13:32:15.114924 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:15.114897    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkwgj_39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e/node-ca/0.log"
Sep 10 13:32:15.910537 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:15.910494    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/egress-router-binary-copy/0.log"
Sep 10 13:32:16.111902 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:16.111874    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/cni-plugins/0.log"
Sep 10 13:32:16.311543 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:16.311456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/bond-cni-plugin/0.log"
Sep 10 13:32:16.510925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:16.510892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/routeoverride-cni/0.log"
Sep 10 13:32:16.713433 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:16.713370    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/whereabouts-cni-bincopy/0.log"
Sep 10 13:32:16.916589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:16.916548    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/whereabouts-cni/0.log"
Sep 10 13:32:17.116294 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:17.116273    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7dzc2_d0e17b82-b1f5-491b-93ea-46303a0c05c9/kube-multus-additional-cni-plugins/0.log"
Sep 10 13:32:20.035708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:20.035678    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-48.ec2.internal" event="NodeReady"
Sep 10 13:32:20.509545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:20.509518    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:32:20.709450 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:20.709418    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/3.log"
Sep 10 13:32:22.512791 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:22.512755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/ovn-controller/0.log"
Sep 10 13:32:22.709869 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:22.709838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/ovn-acl-logging/0.log"
Sep 10 13:32:22.910876 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:22.910824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/kube-rbac-proxy-node/0.log"
Sep 10 13:32:23.111479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:23.111449    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:32:23.310367 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:23.310287    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/northd/0.log"
Sep 10 13:32:23.511835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:23.511804    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/nbdb/0.log"
Sep 10 13:32:23.710403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:23.710378    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/sbdb/0.log"
Sep 10 13:32:23.910405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:23.910376    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g8ln9_602b6dbd-a766-4674-bc7c-5c49b425b045/ovnkube-controller/0.log"
Sep 10 13:32:28.509745 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:28.509718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gxwt2_f8dea817-8747-48a0-9536-7d5df8dec6d2/iptables-alerter/0.log"
Sep 10 13:32:37.109734 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.109700    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"]
Sep 10 13:32:37.111569 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.111553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:37.113951 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.113927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rxsl9\""
Sep 10 13:32:37.114078 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.114022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 13:32:37.133086 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.133064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"]
Sep 10 13:32:37.206581 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.206547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2960223e-bcff-4d5c-8f04-2abde821af3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vg7rr\" (UID: \"2960223e-bcff-4d5c-8f04-2abde821af3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:37.217130 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.217093    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-skb2d"]
Sep 10 13:32:37.219354 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.219329    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f4849c767-t627c"]
Sep 10 13:32:37.219467 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.219452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.221330 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.221299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.221866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.221849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 13:32:37.221962 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.221910    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 13:32:37.221962 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.221935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m68qf\""
Sep 10 13:32:37.223498 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.223482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 13:32:37.223796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.223781    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 13:32:37.223856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.223796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 13:32:37.223912 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.223784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rtcfw\""
Sep 10 13:32:37.230832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.230796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 13:32:37.237151 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.237131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-skb2d"]
Sep 10 13:32:37.239566 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.239527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4849c767-t627c"]
Sep 10 13:32:37.245410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.245392    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-84ts4"]
Sep 10 13:32:37.247265 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.247248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.249463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.249447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 13:32:37.249766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.249746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n9gkj\""
Sep 10 13:32:37.249864 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.249773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 13:32:37.249864 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.249776    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 13:32:37.249864 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.249780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 13:32:37.268077 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.268053    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b2vwx"]
Sep 10 13:32:37.270050 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.270030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84ts4"]
Sep 10 13:32:37.270151 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.270135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.271986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.271968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 13:32:37.272149 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.272086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 13:32:37.272233 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.272214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 13:32:37.272404 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.272381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xxdkm\""
Sep 10 13:32:37.289940 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.289920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2vwx"]
Sep 10 13:32:37.307063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-tls\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f15e82-4927-42fc-afa5-baa414a87a5b-config-volume\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.307155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-trusted-ca\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-installation-pull-secrets\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-image-registry-private-configuration\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f15e82-4927-42fc-afa5-baa414a87a5b-metrics-tls\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5l5\" (UniqueName: \"kubernetes.io/projected/b9f15e82-4927-42fc-afa5-baa414a87a5b-kube-api-access-hr5l5\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2960223e-bcff-4d5c-8f04-2abde821af3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vg7rr\" (UID: \"2960223e-bcff-4d5c-8f04-2abde821af3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f15e82-4927-42fc-afa5-baa414a87a5b-tmp-dir\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f640fe27-8902-4eb0-982f-11af5032e789-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-ca-trust-extracted\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-certificates\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f640fe27-8902-4eb0-982f-11af5032e789-crio-socket\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614bc448-cf0b-40b7-8bfe-28664f494c7b-cert\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f640fe27-8902-4eb0-982f-11af5032e789-data-volume\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvkc\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-kube-api-access-hmvkc\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5qw\" (UniqueName: \"kubernetes.io/projected/f640fe27-8902-4eb0-982f-11af5032e789-kube-api-access-gf5qw\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f640fe27-8902-4eb0-982f-11af5032e789-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-bound-sa-token\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.307609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.307551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pw6c\" (UniqueName: \"kubernetes.io/projected/614bc448-cf0b-40b7-8bfe-28664f494c7b-kube-api-access-9pw6c\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.310664 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.310645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2960223e-bcff-4d5c-8f04-2abde821af3e-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vg7rr\" (UID: \"2960223e-bcff-4d5c-8f04-2abde821af3e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:37.408089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f15e82-4927-42fc-afa5-baa414a87a5b-tmp-dir\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.408089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f640fe27-8902-4eb0-982f-11af5032e789-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-ca-trust-extracted\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-certificates\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f640fe27-8902-4eb0-982f-11af5032e789-crio-socket\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614bc448-cf0b-40b7-8bfe-28664f494c7b-cert\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f640fe27-8902-4eb0-982f-11af5032e789-data-volume\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f640fe27-8902-4eb0-982f-11af5032e789-crio-socket\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvkc\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-kube-api-access-hmvkc\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5qw\" (UniqueName: \"kubernetes.io/projected/f640fe27-8902-4eb0-982f-11af5032e789-kube-api-access-gf5qw\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f640fe27-8902-4eb0-982f-11af5032e789-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-bound-sa-token\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pw6c\" (UniqueName: \"kubernetes.io/projected/614bc448-cf0b-40b7-8bfe-28664f494c7b-kube-api-access-9pw6c\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-tls\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f15e82-4927-42fc-afa5-baa414a87a5b-tmp-dir\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f15e82-4927-42fc-afa5-baa414a87a5b-config-volume\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-trusted-ca\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f640fe27-8902-4eb0-982f-11af5032e789-data-volume\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-installation-pull-secrets\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-image-registry-private-configuration\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f15e82-4927-42fc-afa5-baa414a87a5b-metrics-tls\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.408708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.408711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5l5\" (UniqueName: \"kubernetes.io/projected/b9f15e82-4927-42fc-afa5-baa414a87a5b-kube-api-access-hr5l5\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.409402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.409143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-certificates\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.409402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.409162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f640fe27-8902-4eb0-982f-11af5032e789-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.409402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.409239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b9f15e82-4927-42fc-afa5-baa414a87a5b-config-volume\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.409580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.409466    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-trusted-ca\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.409799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.409773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-ca-trust-extracted\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.411169 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.411142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f640fe27-8902-4eb0-982f-11af5032e789-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.411273 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.411167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-image-registry-private-configuration\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.411809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.411787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/614bc448-cf0b-40b7-8bfe-28664f494c7b-cert\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.411927 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.411908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-installation-pull-secrets\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.411977 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.411907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9f15e82-4927-42fc-afa5-baa414a87a5b-metrics-tls\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.412018 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.412002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-registry-tls\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.419849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.419832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:37.425559 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.425539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5l5\" (UniqueName: \"kubernetes.io/projected/b9f15e82-4927-42fc-afa5-baa414a87a5b-kube-api-access-hr5l5\") pod \"dns-default-skb2d\" (UID: \"b9f15e82-4927-42fc-afa5-baa414a87a5b\") " pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.425779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.425761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-bound-sa-token\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.425961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.425943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pw6c\" (UniqueName: \"kubernetes.io/projected/614bc448-cf0b-40b7-8bfe-28664f494c7b-kube-api-access-9pw6c\") pod \"ingress-canary-b2vwx\" (UID: \"614bc448-cf0b-40b7-8bfe-28664f494c7b\") " pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.426066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.426049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvkc\" (UniqueName: \"kubernetes.io/projected/8f80ae62-51b3-4e45-8b28-a8c7b74da82a-kube-api-access-hmvkc\") pod \"image-registry-7f4849c767-t627c\" (UID: \"8f80ae62-51b3-4e45-8b28-a8c7b74da82a\") " pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.426302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.426285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5qw\" (UniqueName: \"kubernetes.io/projected/f640fe27-8902-4eb0-982f-11af5032e789-kube-api-access-gf5qw\") pod \"insights-runtime-extractor-84ts4\" (UID: \"f640fe27-8902-4eb0-982f-11af5032e789\") " pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.529719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.529690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:37.534300 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.534281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:37.555938 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.555911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84ts4"
Sep 10 13:32:37.577922 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.577891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b2vwx"
Sep 10 13:32:37.699130 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.699057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"]
Sep 10 13:32:37.702227 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:37.702195    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2960223e_bcff_4d5c_8f04_2abde821af3e.slice/crio-c28576354bcf81da7fc78f8af6027d2ab40505dbc4dc0fd069b1363bc9f87562 WatchSource:0}: Error finding container c28576354bcf81da7fc78f8af6027d2ab40505dbc4dc0fd069b1363bc9f87562: Status 404 returned error can't find the container with id c28576354bcf81da7fc78f8af6027d2ab40505dbc4dc0fd069b1363bc9f87562
Sep 10 13:32:37.769332 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.769298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr" event={"ID":"2960223e-bcff-4d5c-8f04-2abde821af3e","Type":"ContainerStarted","Data":"c28576354bcf81da7fc78f8af6027d2ab40505dbc4dc0fd069b1363bc9f87562"}
Sep 10 13:32:37.808481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.808450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-skb2d"]
Sep 10 13:32:37.811911 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.811888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f4849c767-t627c"]
Sep 10 13:32:37.812338 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:37.812307    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f15e82_4927_42fc_afa5_baa414a87a5b.slice/crio-442c7b6d45d0a3f858cf01a23da75c2e4832e3b749f93a7c4cb3fa809c9a97b8 WatchSource:0}: Error finding container 442c7b6d45d0a3f858cf01a23da75c2e4832e3b749f93a7c4cb3fa809c9a97b8: Status 404 returned error can't find the container with id 442c7b6d45d0a3f858cf01a23da75c2e4832e3b749f93a7c4cb3fa809c9a97b8
Sep 10 13:32:37.817000 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:37.816978    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f80ae62_51b3_4e45_8b28_a8c7b74da82a.slice/crio-f403e4360667cf3a0697d9fc5f2aab171b625ac4c13fba51b8bd406cb837f6f6 WatchSource:0}: Error finding container f403e4360667cf3a0697d9fc5f2aab171b625ac4c13fba51b8bd406cb837f6f6: Status 404 returned error can't find the container with id f403e4360667cf3a0697d9fc5f2aab171b625ac4c13fba51b8bd406cb837f6f6
Sep 10 13:32:37.857177 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.856461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84ts4"]
Sep 10 13:32:37.860051 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:37.860027    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf640fe27_8902_4eb0_982f_11af5032e789.slice/crio-2212261ce0617d431da84adc693f30f71e54ec563ae7160da72c6faefe33b876 WatchSource:0}: Error finding container 2212261ce0617d431da84adc693f30f71e54ec563ae7160da72c6faefe33b876: Status 404 returned error can't find the container with id 2212261ce0617d431da84adc693f30f71e54ec563ae7160da72c6faefe33b876
Sep 10 13:32:37.876119 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:37.876097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b2vwx"]
Sep 10 13:32:37.879355 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:37.879330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614bc448_cf0b_40b7_8bfe_28664f494c7b.slice/crio-b8e70f7b2b90e7e91cae2cf84c24c42d7e4ca3e810dc2a0903a880cca1467be6 WatchSource:0}: Error finding container b8e70f7b2b90e7e91cae2cf84c24c42d7e4ca3e810dc2a0903a880cca1467be6: Status 404 returned error can't find the container with id b8e70f7b2b90e7e91cae2cf84c24c42d7e4ca3e810dc2a0903a880cca1467be6
Sep 10 13:32:38.774524 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.774472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2vwx" event={"ID":"614bc448-cf0b-40b7-8bfe-28664f494c7b","Type":"ContainerStarted","Data":"b8e70f7b2b90e7e91cae2cf84c24c42d7e4ca3e810dc2a0903a880cca1467be6"}
Sep 10 13:32:38.776874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.776850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84ts4" event={"ID":"f640fe27-8902-4eb0-982f-11af5032e789","Type":"ContainerStarted","Data":"a073346eb8cd40d35e6687435c40d2e9aa62bb9afa13247d4b4957b994c02c17"}
Sep 10 13:32:38.776982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.776905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84ts4" event={"ID":"f640fe27-8902-4eb0-982f-11af5032e789","Type":"ContainerStarted","Data":"113c22a5f4c7cfd73832d299f49657c3613f28513c09bf647a4cb637cafcd867"}
Sep 10 13:32:38.776982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.776934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84ts4" event={"ID":"f640fe27-8902-4eb0-982f-11af5032e789","Type":"ContainerStarted","Data":"2212261ce0617d431da84adc693f30f71e54ec563ae7160da72c6faefe33b876"}
Sep 10 13:32:38.779263 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.779228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-skb2d" event={"ID":"b9f15e82-4927-42fc-afa5-baa414a87a5b","Type":"ContainerStarted","Data":"442c7b6d45d0a3f858cf01a23da75c2e4832e3b749f93a7c4cb3fa809c9a97b8"}
Sep 10 13:32:38.781059 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.781038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4849c767-t627c" event={"ID":"8f80ae62-51b3-4e45-8b28-a8c7b74da82a","Type":"ContainerStarted","Data":"cdcf03c546c5e48cd300617576cb59442d4d6a7270b2ae7e09e34bcf696bd2d2"}
Sep 10 13:32:38.781139 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.781076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f4849c767-t627c" event={"ID":"8f80ae62-51b3-4e45-8b28-a8c7b74da82a","Type":"ContainerStarted","Data":"f403e4360667cf3a0697d9fc5f2aab171b625ac4c13fba51b8bd406cb837f6f6"}
Sep 10 13:32:38.781215 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:38.781201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:32:39.786241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:39.786211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-skb2d" event={"ID":"b9f15e82-4927-42fc-afa5-baa414a87a5b","Type":"ContainerStarted","Data":"a68929dce9b110a3dc23a5356b41a12b51cfad616413211d1e2fc877d9dbd8d4"}
Sep 10 13:32:39.788007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:39.787980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr" event={"ID":"2960223e-bcff-4d5c-8f04-2abde821af3e","Type":"ContainerStarted","Data":"0068b6c6cd1e107377feb7055db4739b4100e1322053b7fcc338fa392eb66004"}
Sep 10 13:32:39.809850 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:39.809791    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr" podStartSLOduration=0.899446791 podStartE2EDuration="2.809770937s" totalImagesPullingTime="1.910324146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:39.808624747 +0000 UTC m=+244.263499933" watchObservedRunningTime="2026-09-10 13:32:39.809770937 +0000 UTC m=+244.264646103"
Sep 10 13:32:39.810298 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:39.810263    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f4849c767-t627c" podStartSLOduration=2.810251344 podStartE2EDuration="2.810251344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:38.809823226 +0000 UTC m=+243.264698393" watchObservedRunningTime="2026-09-10 13:32:39.810251344 +0000 UTC m=+244.265126522"
Sep 10 13:32:40.792894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:40.792860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-skb2d" event={"ID":"b9f15e82-4927-42fc-afa5-baa414a87a5b","Type":"ContainerStarted","Data":"7e9c9f8eb8512d11954c4b13c091210cc9b56f28f9549d25d05e8bc77031e1d2"}
Sep 10 13:32:40.793216 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:40.793046    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:40.793278 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:40.793252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:40.797793 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:40.797772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vg7rr"
Sep 10 13:32:40.817813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:40.817704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-skb2d" podStartSLOduration=2.012471124 podStartE2EDuration="3.81768839s" totalImagesPullingTime="1.805217266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:40.817344599 +0000 UTC m=+245.272219767" watchObservedRunningTime="2026-09-10 13:32:40.81768839 +0000 UTC m=+245.272563555"
Sep 10 13:32:41.798519 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:41.798466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b2vwx" event={"ID":"614bc448-cf0b-40b7-8bfe-28664f494c7b","Type":"ContainerStarted","Data":"6962fbf40f837d109c86bd87b261ff681e67f1e618f53840af396df40f6acc3a"}
Sep 10 13:32:42.804029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:42.803992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84ts4" event={"ID":"f640fe27-8902-4eb0-982f-11af5032e789","Type":"ContainerStarted","Data":"b6918eddb1784be5c4e8038b26d1081cb99b95f938d42519a0a8feaef8680a1d"}
Sep 10 13:32:42.829933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:42.829876    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-84ts4" podStartSLOduration=1.965755076 podStartE2EDuration="5.829863153s" totalImagesPullingTime="3.864108077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:42.829689451 +0000 UTC m=+247.284564643" watchObservedRunningTime="2026-09-10 13:32:42.829863153 +0000 UTC m=+247.284738316"
Sep 10 13:32:42.830072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:42.829969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b2vwx" podStartSLOduration=2.9663904629999998 podStartE2EDuration="5.829964447s" totalImagesPullingTime="2.863573984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:41.832575909 +0000 UTC m=+246.287451075" watchObservedRunningTime="2026-09-10 13:32:42.829964447 +0000 UTC m=+247.284839611"
Sep 10 13:32:46.667053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.667020    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l5gb7"]
Sep 10 13:32:46.671481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.671455    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"]
Sep 10 13:32:46.671623 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.671607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.673339 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.673322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.676094 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.676064    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-l5gb7\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="e2644f21-2f03-459e-b7c0-c766136b6162" pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.676183 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.676076    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 10 13:32:46.676183 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.676103    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret"
Sep 10 13:32:46.676183 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.676103    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap"
Sep 10 13:32:46.677624 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677572    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-state-metrics-custom-resource-state-configmap\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap"
Sep 10 13:32:46.677624 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677601    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:32:46.677624 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677610    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap"
Sep 10 13:32:46.677798 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677615    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-dockercfg-wk9g8\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wk9g8\"" type="*v1.Secret"
Sep 10 13:32:46.677798 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677647    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 10 13:32:46.677798 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677646    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:32:46.677798 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677654    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-s6fhr\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6fhr\"" type="*v1.Secret"
Sep 10 13:32:46.677798 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:46.677669    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret"
Sep 10 13:32:46.691647 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.691628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"]
Sep 10 13:32:46.780596 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9xt\" (UniqueName: \"kubernetes.io/projected/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-api-access-tf9xt\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.780596 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-textfile\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.780773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-wtmp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.780773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjrp\" (UniqueName: \"kubernetes.io/projected/e2644f21-2f03-459e-b7c0-c766136b6162-kube-api-access-5qjrp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dd35db0-2c03-4be2-99ae-8592b784cde1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.780954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.780954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-root\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.780954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.781105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.780989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-sys\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.781105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.781022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.781105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.781041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.881826 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.881798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.881964 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.881839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.881957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9xt\" (UniqueName: \"kubernetes.io/projected/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-api-access-tf9xt\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.882046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.881987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-textfile\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.882046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-wtmp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-wtmp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.882267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjrp\" (UniqueName: \"kubernetes.io/projected/e2644f21-2f03-459e-b7c0-c766136b6162-kube-api-access-5qjrp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dd35db0-2c03-4be2-99ae-8592b784cde1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.882422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:46.882422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-root\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-textfile\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-root\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-sys\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e2644f21-2f03-459e-b7c0-c766136b6162-sys\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:46.882646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:46.882600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dd35db0-2c03-4be2-99ae-8592b784cde1-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:47.632458 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.632336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 13:32:47.635654 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.635620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:47.635958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.635935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wk9g8\""
Sep 10 13:32:47.687436 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.687408    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 13:32:47.692870 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.692850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-accelerators-collector-config\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:47.736577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.736549    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:47.739072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.739049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.743147 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743125    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 13:32:47.743270 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-glqq7\""
Sep 10 13:32:47.743336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 13:32:47.743336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743331    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 13:32:47.743544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 13:32:47.743544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743539    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 13:32:47.743655 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 13:32:47.743787 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743773    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 13:32:47.743849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 13:32:47.743904 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.743860    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 13:32:47.761008 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.760984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 13:32:47.762854 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.762834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:47.789064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789067    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789229    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdwb\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.789585 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.789462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.880850 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.880821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 13:32:47.882587 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882542    2577 configmap.go:193] Couldn't get configMap openshift-monitoring/metrics-client-ca: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882587 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882562    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882591    2577 configmap.go:193] Couldn't get configMap openshift-monitoring/kube-state-metrics-custom-resource-state-configmap: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882602    2577 configmap.go:193] Couldn't get configMap openshift-monitoring/metrics-client-ca: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882632    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls podName:6dd35db0-2c03-4be2-99ae-8592b784cde1 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:48.382618053 +0000 UTC m=+252.837493207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-f7mw5" (UID: "6dd35db0-2c03-4be2-99ae-8592b784cde1") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882658    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca podName:e2644f21-2f03-459e-b7c0-c766136b6162 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:48.382641887 +0000 UTC m=+252.837517028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-client-ca" (UniqueName: "kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca") pod "node-exporter-l5gb7" (UID: "e2644f21-2f03-459e-b7c0-c766136b6162") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882542    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca podName:6dd35db0-2c03-4be2-99ae-8592b784cde1 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:48.382670117 +0000 UTC m=+252.837545259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-client-ca" (UniqueName: "kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca") pod "kube-state-metrics-74cb57df7-f7mw5" (UID: "6dd35db0-2c03-4be2-99ae-8592b784cde1") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap podName:6dd35db0-2c03-4be2-99ae-8592b784cde1 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:48.382686838 +0000 UTC m=+252.837561983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-custom-resource-state-configmap" (UniqueName: "kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap") pod "kube-state-metrics-74cb57df7-f7mw5" (UID: "6dd35db0-2c03-4be2-99ae-8592b784cde1") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:32:47.882953 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:32:47.882727    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls podName:e2644f21-2f03-459e-b7c0-c766136b6162 nodeName:}" failed. No retries permitted until 2026-09-10 13:32:48.382709161 +0000 UTC m=+252.837584315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls") pod "node-exporter-l5gb7" (UID: "e2644f21-2f03-459e-b7c0-c766136b6162") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:32:47.884729 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.884710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:47.889798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.889889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.889889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.889889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.889889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.889996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdwb\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.890459 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.890398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.891232 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.891207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.892643 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.892619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.892747 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.892669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.892747 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.892736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.892954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.892927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.893093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.893071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.893146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.893119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.893303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.893277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.894067 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.894044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.894770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.894752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:47.916553 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.916535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 13:32:47.936336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:47.936315    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6fhr\""
Sep 10 13:32:48.084707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.084677    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 13:32:48.116974 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.116948    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 13:32:48.121293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.121274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:48.184887 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.184829    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 13:32:48.212110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.212087    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 13:32:48.215398 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.215374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjrp\" (UniqueName: \"kubernetes.io/projected/e2644f21-2f03-459e-b7c0-c766136b6162-kube-api-access-5qjrp\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.215501 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.215447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9xt\" (UniqueName: \"kubernetes.io/projected/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-api-access-tf9xt\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.216874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.216857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdwb\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb\") pod \"alertmanager-main-0\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:48.348267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.348219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:32:48.394411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.394191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.394411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.394225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.394411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.394251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.394411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.394305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.394411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.394337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.395080 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.395049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2644f21-2f03-459e-b7c0-c766136b6162-metrics-client-ca\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.395430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.395408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.395659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.395458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.397673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.397476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e2644f21-2f03-459e-b7c0-c766136b6162-node-exporter-tls\") pod \"node-exporter-l5gb7\" (UID: \"e2644f21-2f03-459e-b7c0-c766136b6162\") " pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.397673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.397628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dd35db0-2c03-4be2-99ae-8592b784cde1-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-f7mw5\" (UID: \"6dd35db0-2c03-4be2-99ae-8592b784cde1\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.481770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.481691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l5gb7"
Sep 10 13:32:48.486369 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.486350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"
Sep 10 13:32:48.492982 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:48.492960    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2644f21_2f03_459e_b7c0_c766136b6162.slice/crio-f0c9489c2ac67d35bf033d05bdef547b093f06e58784c5b7c7dcd9e4870bf4c0 WatchSource:0}: Error finding container f0c9489c2ac67d35bf033d05bdef547b093f06e58784c5b7c7dcd9e4870bf4c0: Status 404 returned error can't find the container with id f0c9489c2ac67d35bf033d05bdef547b093f06e58784c5b7c7dcd9e4870bf4c0
Sep 10 13:32:48.640402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.640368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:32:48.644325 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:48.644299    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d222c5_8048_420e_af3e_f18f0d1f630f.slice/crio-0210e3224410c93e526b52b4956009e695ca708a306293f56ff0061265bb993b WatchSource:0}: Error finding container 0210e3224410c93e526b52b4956009e695ca708a306293f56ff0061265bb993b: Status 404 returned error can't find the container with id 0210e3224410c93e526b52b4956009e695ca708a306293f56ff0061265bb993b
Sep 10 13:32:48.766608 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.766540    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5"]
Sep 10 13:32:48.769779 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:48.769755    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd35db0_2c03_4be2_99ae_8592b784cde1.slice/crio-998cc0a5ae0b304352ebc5ecd88395c3f0e4629e318e27f4f5bba207c9ac830b WatchSource:0}: Error finding container 998cc0a5ae0b304352ebc5ecd88395c3f0e4629e318e27f4f5bba207c9ac830b: Status 404 returned error can't find the container with id 998cc0a5ae0b304352ebc5ecd88395c3f0e4629e318e27f4f5bba207c9ac830b
Sep 10 13:32:48.823948 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.823918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5" event={"ID":"6dd35db0-2c03-4be2-99ae-8592b784cde1","Type":"ContainerStarted","Data":"998cc0a5ae0b304352ebc5ecd88395c3f0e4629e318e27f4f5bba207c9ac830b"}
Sep 10 13:32:48.825055 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.825032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5gb7" event={"ID":"e2644f21-2f03-459e-b7c0-c766136b6162","Type":"ContainerStarted","Data":"f0c9489c2ac67d35bf033d05bdef547b093f06e58784c5b7c7dcd9e4870bf4c0"}
Sep 10 13:32:48.826082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:48.826063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"0210e3224410c93e526b52b4956009e695ca708a306293f56ff0061265bb993b"}
Sep 10 13:32:49.831157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:49.831065    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2644f21-2f03-459e-b7c0-c766136b6162" containerID="bee7c1c46b07682c2b6a3afd54d0b1737e47ce3fd0be7ed7b68b7c59390ed5d0" exitCode=0
Sep 10 13:32:49.831157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:49.831106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5gb7" event={"ID":"e2644f21-2f03-459e-b7c0-c766136b6162","Type":"ContainerDied","Data":"bee7c1c46b07682c2b6a3afd54d0b1737e47ce3fd0be7ed7b68b7c59390ed5d0"}
Sep 10 13:32:50.801930 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.801850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-skb2d"
Sep 10 13:32:50.839698 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.839144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5" event={"ID":"6dd35db0-2c03-4be2-99ae-8592b784cde1","Type":"ContainerStarted","Data":"d38b2aa79f853618634955e96de4ddbc4d232f4588cf02f7a0542a75f302211a"}
Sep 10 13:32:50.839698 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.839215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5" event={"ID":"6dd35db0-2c03-4be2-99ae-8592b784cde1","Type":"ContainerStarted","Data":"0b9bbf3c99cb9fe65f211a1fec06085d2c21ed24c3907d147318712d39702cf1"}
Sep 10 13:32:50.839698 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.839246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5" event={"ID":"6dd35db0-2c03-4be2-99ae-8592b784cde1","Type":"ContainerStarted","Data":"e2d538bef1afb2b62f9556e94a88be3a58d3d11013d5d30752b7ccb50a0980e1"}
Sep 10 13:32:50.842094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.842061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5gb7" event={"ID":"e2644f21-2f03-459e-b7c0-c766136b6162","Type":"ContainerStarted","Data":"c7b49a13a4e3faa1d1c7da2bbabb512eac97e70808466b2608128c55fa20a985"}
Sep 10 13:32:50.842255 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.842114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l5gb7" event={"ID":"e2644f21-2f03-459e-b7c0-c766136b6162","Type":"ContainerStarted","Data":"7539a6e3af1c59023ad16e2d43399a1db6142d2645b1cad77aab988bcf49b5e3"}
Sep 10 13:32:50.863407 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.863350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-f7mw5" podStartSLOduration=3.227290841 podStartE2EDuration="4.863327007s" totalImagesPullingTime="1.636036166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:50.862588878 +0000 UTC m=+255.317464043" watchObservedRunningTime="2026-09-10 13:32:50.863327007 +0000 UTC m=+255.318202172"
Sep 10 13:32:50.882531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:50.882473    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l5gb7" podStartSLOduration=3.806449629 podStartE2EDuration="4.882455841s" totalImagesPullingTime="1.014832182s" totalInitContainerRuntime="61.17403ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:50.882034865 +0000 UTC m=+255.336910030" watchObservedRunningTime="2026-09-10 13:32:50.882455841 +0000 UTC m=+255.337331003"
Sep 10 13:32:51.846382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:51.846349    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f" exitCode=0
Sep 10 13:32:51.846774 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:51.846437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f"}
Sep 10 13:32:53.080995 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.080961    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:53.083813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.083790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.087203 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.087179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 13:32:53.087308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.087180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 13:32:53.087308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.087299    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 13:32:53.088693 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.088667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 13:32:53.088805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.088732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 13:32:53.088848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.088800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b01javqsmojgo\""
Sep 10 13:32:53.088963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.088949    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 13:32:53.089549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.089335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 13:32:53.089549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.089373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 13:32:53.089549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.089482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 13:32:53.089946 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.089886    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 13:32:53.090075 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.089947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 13:32:53.090165 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.090140    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 13:32:53.090379 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.090236    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 13:32:53.090379 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.090296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmdnc\""
Sep 10 13:32:53.091946 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.091925    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 13:32:53.094703 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.094680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 13:32:53.118040 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.118010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:53.235910 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.235871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.235923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h94sd\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.236988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.236662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.337755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338010 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.337982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h94sd\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.338803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.338755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.339536 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.339493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.340361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.339985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.340361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.339993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.340361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.340302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.341523 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.341483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.342290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.342265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.343600 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.343577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.343741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344629 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.344741 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.344631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.345769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.345748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.349825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.349719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h94sd\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd\") pod \"prometheus-k8s-0\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.395371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.395345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:32:53.688721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.688693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:32:53.691676 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:32:53.691655    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf44d65_5eab_400a_b5be_0ffdc3c1d485.slice/crio-ae1263a7301aa460b967fc0cc912f0037c036ae20248a16150892185d1741f8c WatchSource:0}: Error finding container ae1263a7301aa460b967fc0cc912f0037c036ae20248a16150892185d1741f8c: Status 404 returned error can't find the container with id ae1263a7301aa460b967fc0cc912f0037c036ae20248a16150892185d1741f8c
Sep 10 13:32:53.856797 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.856706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"}
Sep 10 13:32:53.856797 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.856788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"}
Sep 10 13:32:53.856970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.856833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"}
Sep 10 13:32:53.856970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.856863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"}
Sep 10 13:32:53.856970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.856888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"}
Sep 10 13:32:53.858066 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.858042    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3" exitCode=0
Sep 10 13:32:53.858177 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.858133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3"}
Sep 10 13:32:53.858227 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:53.858188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"ae1263a7301aa460b967fc0cc912f0037c036ae20248a16150892185d1741f8c"}
Sep 10 13:32:54.866261 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:54.866220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerStarted","Data":"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"}
Sep 10 13:32:54.919738 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:54.919688    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.008145358 podStartE2EDuration="7.919665801s" totalImagesPullingTime="4.895405107s" totalInitContainerRuntime="16.115336ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:54.918956241 +0000 UTC m=+259.373831410" watchObservedRunningTime="2026-09-10 13:32:54.919665801 +0000 UTC m=+259.374540966"
Sep 10 13:32:56.877080 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:56.877041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"}
Sep 10 13:32:56.877463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:56.877156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"}
Sep 10 13:32:59.793773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:32:59.793634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f4849c767-t627c"
Sep 10 13:33:00.894289 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:00.894253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"}
Sep 10 13:33:00.894659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:00.894314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"}
Sep 10 13:33:00.894659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:00.894339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"}
Sep 10 13:33:00.894659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:00.894364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerStarted","Data":"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"}
Sep 10 13:33:00.925497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:00.925438    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.010816128 podStartE2EDuration="7.925419172s" totalImagesPullingTime="5.896966598s" totalInitContainerRuntime="17.636446ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:00.923904538 +0000 UTC m=+265.378779712" watchObservedRunningTime="2026-09-10 13:33:00.925419172 +0000 UTC m=+265.380294335"
Sep 10 13:33:03.396286 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:03.396258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:04.040473 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.040425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:33:04.042808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.042790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 13:33:04.052700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.052678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13a758ab-ae02-4484-b6f1-0773f7e2a50a-original-pull-secret\") pod \"global-pull-secret-syncer-9slnm\" (UID: \"13a758ab-ae02-4484-b6f1-0773f7e2a50a\") " pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:33:04.176819 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.176788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9slnm"
Sep 10 13:33:04.446768 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.446743    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9slnm"]
Sep 10 13:33:04.449225 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:33:04.449200    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a758ab_ae02_4484_b6f1_0773f7e2a50a.slice/crio-5a3763a070860406e827b0c63fef646c6cf6857ff4a543d7af1667c8a7d4bced WatchSource:0}: Error finding container 5a3763a070860406e827b0c63fef646c6cf6857ff4a543d7af1667c8a7d4bced: Status 404 returned error can't find the container with id 5a3763a070860406e827b0c63fef646c6cf6857ff4a543d7af1667c8a7d4bced
Sep 10 13:33:04.915787 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:04.915748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9slnm" event={"ID":"13a758ab-ae02-4484-b6f1-0773f7e2a50a","Type":"ContainerStarted","Data":"5a3763a070860406e827b0c63fef646c6cf6857ff4a543d7af1667c8a7d4bced"}
Sep 10 13:33:05.856350 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.856314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:33:05.859597 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.859572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 13:33:05.869355 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.869336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9160df28-24ae-4948-af5b-3b8b37a617f1-metrics-certs\") pod \"network-metrics-daemon-t4xx4\" (UID: \"9160df28-24ae-4948-af5b-3b8b37a617f1\") " pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:33:05.957251 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.957218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:33:05.960633 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.960611    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 13:33:05.969303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.969283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4vqdt\""
Sep 10 13:33:05.970070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.970045    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 13:33:05.977976 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.977956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4xx4"
Sep 10 13:33:05.981717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:05.981697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296zg\" (UniqueName: \"kubernetes.io/projected/342e788e-fc18-4211-b510-0be9a5751ba1-kube-api-access-296zg\") pod \"network-check-target-fzppt\" (UID: \"342e788e-fc18-4211-b510-0be9a5751ba1\") " pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:33:06.273292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.273238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4xx4"]
Sep 10 13:33:06.274881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.274858    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cdbpg\""
Sep 10 13:33:06.275720 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:33:06.275693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9160df28_24ae_4948_af5b_3b8b37a617f1.slice/crio-3a5c35a9614e5a7a3c2af28c441d37ef6b166efa07f0008f84ac5f837ffe1e16 WatchSource:0}: Error finding container 3a5c35a9614e5a7a3c2af28c441d37ef6b166efa07f0008f84ac5f837ffe1e16: Status 404 returned error can't find the container with id 3a5c35a9614e5a7a3c2af28c441d37ef6b166efa07f0008f84ac5f837ffe1e16
Sep 10 13:33:06.283177 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.283159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:33:06.577673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.577642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fzppt"]
Sep 10 13:33:06.579055 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:33:06.579029    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342e788e_fc18_4211_b510_0be9a5751ba1.slice/crio-499f41478a86746a69cd47120c4ab63582c734bbf116285805279b9c1c00a098 WatchSource:0}: Error finding container 499f41478a86746a69cd47120c4ab63582c734bbf116285805279b9c1c00a098: Status 404 returned error can't find the container with id 499f41478a86746a69cd47120c4ab63582c734bbf116285805279b9c1c00a098
Sep 10 13:33:06.927206 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.927166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4xx4" event={"ID":"9160df28-24ae-4948-af5b-3b8b37a617f1","Type":"ContainerStarted","Data":"3a5c35a9614e5a7a3c2af28c441d37ef6b166efa07f0008f84ac5f837ffe1e16"}
Sep 10 13:33:06.928577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:06.928540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fzppt" event={"ID":"342e788e-fc18-4211-b510-0be9a5751ba1","Type":"ContainerStarted","Data":"499f41478a86746a69cd47120c4ab63582c734bbf116285805279b9c1c00a098"}
Sep 10 13:33:10.945099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.945059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4xx4" event={"ID":"9160df28-24ae-4948-af5b-3b8b37a617f1","Type":"ContainerStarted","Data":"9e14ec4ff4caa6213af166bf86127e8b4b757d935522376d31b4aca4328255ed"}
Sep 10 13:33:10.945572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.945120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4xx4" event={"ID":"9160df28-24ae-4948-af5b-3b8b37a617f1","Type":"ContainerStarted","Data":"821cfe5a0f9acb8e17adca575e394ebfdb8fb16c5f52a41fb6afcd88cce110e4"}
Sep 10 13:33:10.946623 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.946598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fzppt" event={"ID":"342e788e-fc18-4211-b510-0be9a5751ba1","Type":"ContainerStarted","Data":"3245db6252346afaab3f90ec4b072898efc9a98cba8c9a34b26660f755e1c99d"}
Sep 10 13:33:10.946738 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.946720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:33:10.947965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.947944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9slnm" event={"ID":"13a758ab-ae02-4484-b6f1-0773f7e2a50a","Type":"ContainerStarted","Data":"e7dd8e5b5e6d8291b1110dfb25855652bd5a3e19ddaafbb3c01a1c9b3c675c5d"}
Sep 10 13:33:10.970334 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.970292    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t4xx4" podStartSLOduration=252.337366841 podStartE2EDuration="4m15.970278668s" totalImagesPullingTime="3.632911827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:10.969526742 +0000 UTC m=+275.424401921" watchObservedRunningTime="2026-09-10 13:33:10.970278668 +0000 UTC m=+275.425153868"
Sep 10 13:33:10.993725 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:10.993686    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9slnm" podStartSLOduration=129.519913182 podStartE2EDuration="2m14.993670772s" totalImagesPullingTime="5.47375759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:10.993397368 +0000 UTC m=+275.448272532" watchObservedRunningTime="2026-09-10 13:33:10.993670772 +0000 UTC m=+275.448545949"
Sep 10 13:33:11.019003 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:11.018966    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fzppt" podStartSLOduration=252.675732924 podStartE2EDuration="4m16.018954515s" totalImagesPullingTime="3.343221591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:11.018003061 +0000 UTC m=+275.472878235" watchObservedRunningTime="2026-09-10 13:33:11.018954515 +0000 UTC m=+275.473829676"
Sep 10 13:33:25.201275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.201237    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"]
Sep 10 13:33:25.204440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.204421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.207690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.207671    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 13:33:25.208544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.208530    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 13:33:25.209717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.209689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 13:33:25.209717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.209694    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 13:33:25.211330 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.211315    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 13:33:25.213665 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.213650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 13:33:25.213733 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.213687    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 13:33:25.219462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.219442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"]
Sep 10 13:33:25.310625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtlx4\" (UniqueName: \"kubernetes.io/projected/ab24e9b4-80ba-499d-a559-95b55dbf048e-kube-api-access-jtlx4\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.310625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.310853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.310853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.310853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.310853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.310746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab24e9b4-80ba-499d-a559-95b55dbf048e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412067 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab24e9b4-80ba-499d-a559-95b55dbf048e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtlx4\" (UniqueName: \"kubernetes.io/projected/ab24e9b4-80ba-499d-a559-95b55dbf048e-kube-api-access-jtlx4\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.412884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.412853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab24e9b4-80ba-499d-a559-95b55dbf048e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.414849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.414829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-ca\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.414968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.414949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.415013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.414964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.415089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.415069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ab24e9b4-80ba-499d-a559-95b55dbf048e-hub\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.422673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.422651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtlx4\" (UniqueName: \"kubernetes.io/projected/ab24e9b4-80ba-499d-a559-95b55dbf048e-kube-api-access-jtlx4\") pod \"cluster-proxy-proxy-agent-6559895fcd-jjvgk\" (UID: \"ab24e9b4-80ba-499d-a559-95b55dbf048e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.529578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.529465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"
Sep 10 13:33:25.800211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:25.800137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk"]
Sep 10 13:33:25.803073 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:33:25.803043    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab24e9b4_80ba_499d_a559_95b55dbf048e.slice/crio-b6abaadd99971b0f15b5a62f63a8204f31657d592c580b87a173133e2b767ab3 WatchSource:0}: Error finding container b6abaadd99971b0f15b5a62f63a8204f31657d592c580b87a173133e2b767ab3: Status 404 returned error can't find the container with id b6abaadd99971b0f15b5a62f63a8204f31657d592c580b87a173133e2b767ab3
Sep 10 13:33:26.006422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:26.006386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk" event={"ID":"ab24e9b4-80ba-499d-a559-95b55dbf048e","Type":"ContainerStarted","Data":"b6abaadd99971b0f15b5a62f63a8204f31657d592c580b87a173133e2b767ab3"}
Sep 10 13:33:30.028248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:30.028211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk" event={"ID":"ab24e9b4-80ba-499d-a559-95b55dbf048e","Type":"ContainerStarted","Data":"317c6c44d106e076c547f9ff3a008abe21be351c549de30d0b6ee18f94114957"}
Sep 10 13:33:30.028248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:30.028260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk" event={"ID":"ab24e9b4-80ba-499d-a559-95b55dbf048e","Type":"ContainerStarted","Data":"6e6369f4d401face7ed5da9ddcb3e26e854812eb98e414d90ffa735e22ce0aa8"}
Sep 10 13:33:30.028881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:30.028274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk" event={"ID":"ab24e9b4-80ba-499d-a559-95b55dbf048e","Type":"ContainerStarted","Data":"7fd63b77312040fac24b7a57843d5567c30709cfae200ab1ee69c8ebb4da0118"}
Sep 10 13:33:30.053878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:30.053770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6559895fcd-jjvgk" podStartSLOduration=1.37362069 podStartE2EDuration="5.053750394s" totalImagesPullingTime="3.680129704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:33:30.052960757 +0000 UTC m=+294.507835919" watchObservedRunningTime="2026-09-10 13:33:30.053750394 +0000 UTC m=+294.508625555"
Sep 10 13:33:35.977541 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:35.977497    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:33:35.981052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:35.981029    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:33:35.983053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:35.983033    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 13:33:41.954304 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:41.954272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fzppt"
Sep 10 13:33:53.396458 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:53.396382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:53.426906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:53.426874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:33:54.131436 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:33:54.131402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:07.192196 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192159    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192451    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="alertmanager" containerID="cri-o://b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be" gracePeriod=120
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192478    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-web" containerID="cri-o://b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c" gracePeriod=120
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192458    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy" containerID="cri-o://f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7" gracePeriod=120
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192531    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-metric" containerID="cri-o://7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9" gracePeriod=120
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192538    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="prom-label-proxy" containerID="cri-o://ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6" gracePeriod=120
Sep 10 13:34:07.192757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:07.192541    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="config-reloader" containerID="cri-o://b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557" gracePeriod=120
Sep 10 13:34:08.179627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179592    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6" exitCode=0
Sep 10 13:34:08.179627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179619    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9" exitCode=0
Sep 10 13:34:08.179627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179625    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7" exitCode=0
Sep 10 13:34:08.179627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179632    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557" exitCode=0
Sep 10 13:34:08.179627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179637    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be" exitCode=0
Sep 10 13:34:08.179913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"}
Sep 10 13:34:08.179913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"}
Sep 10 13:34:08.179913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"}
Sep 10 13:34:08.179913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"}
Sep 10 13:34:08.179913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.179817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"}
Sep 10 13:34:08.434161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.434106    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:34:08.465390 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465485    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prdwb\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.465811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.465784    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy\") pod \"11d222c5-8048-420e-af3e-f18f0d1f630f\" (UID: \"11d222c5-8048-420e-af3e-f18f0d1f630f\") "
Sep 10 13:34:08.466809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.466413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:34:08.466809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.466633    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:08.467050 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.466909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:08.468976 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.468941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.469129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.469092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume" (OuterVolumeSpecName: "config-volume") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.469476 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.469316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.469476 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.469450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.473785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.473746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:34:08.474261 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.474121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out" (OuterVolumeSpecName: "config-out") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:34:08.474472 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.474401    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.474589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.474493    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb" (OuterVolumeSpecName: "kube-api-access-prdwb") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "kube-api-access-prdwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:34:08.476857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.476833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.482903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.482879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config" (OuterVolumeSpecName: "web-config") pod "11d222c5-8048-420e-af3e-f18f0d1f630f" (UID: "11d222c5-8048-420e-af3e-f18f0d1f630f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:08.566373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566342    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-main-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566365    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-main-db\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566396    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-web-config\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566405    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-metrics-client-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566415    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-tls-assets\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566424    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566434    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11d222c5-8048-420e-af3e-f18f0d1f630f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566443    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdwb\" (UniqueName: \"kubernetes.io/projected/11d222c5-8048-420e-af3e-f18f0d1f630f-kube-api-access-prdwb\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566451    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-cluster-tls-config\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566459    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11d222c5-8048-420e-af3e-f18f0d1f630f-config-out\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566467    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566475    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:08.566571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:08.566483    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11d222c5-8048-420e-af3e-f18f0d1f630f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:09.188729 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.188695    2577 generic.go:356] "Generic (PLEG): container finished" podID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerID="b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c" exitCode=0
Sep 10 13:34:09.188897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.188749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"}
Sep 10 13:34:09.188897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.188792    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 13:34:09.188897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.188831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"11d222c5-8048-420e-af3e-f18f0d1f630f","Type":"ContainerDied","Data":"0210e3224410c93e526b52b4956009e695ca708a306293f56ff0061265bb993b"}
Sep 10 13:34:09.188897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.188879    2577 scope.go:117] "RemoveContainer" containerID="ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"
Sep 10 13:34:09.196209 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.196184    2577 scope.go:117] "RemoveContainer" containerID="7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"
Sep 10 13:34:09.203492 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.203475    2577 scope.go:117] "RemoveContainer" containerID="f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"
Sep 10 13:34:09.209855 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.209838    2577 scope.go:117] "RemoveContainer" containerID="b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"
Sep 10 13:34:09.215809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.215793    2577 scope.go:117] "RemoveContainer" containerID="b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"
Sep 10 13:34:09.223097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.223074    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:34:09.223922 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.223888    2577 scope.go:117] "RemoveContainer" containerID="b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"
Sep 10 13:34:09.224963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.224902    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 13:34:09.230752 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.230735    2577 scope.go:117] "RemoveContainer" containerID="98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f"
Sep 10 13:34:09.237210 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.237193    2577 scope.go:117] "RemoveContainer" containerID="ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"
Sep 10 13:34:09.237471 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.237454    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6\": container with ID starting with ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6 not found: ID does not exist" containerID="ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"
Sep 10 13:34:09.237544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.237477    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6"} err="failed to get container status \"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6\": rpc error: code = NotFound desc = could not find container \"ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6\": container with ID starting with ab013b8cafc8df8187960245118096d1ca59ea2072508f704857391f823395d6 not found: ID does not exist"
Sep 10 13:34:09.237544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.237528    2577 scope.go:117] "RemoveContainer" containerID="7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"
Sep 10 13:34:09.237779 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.237761    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9\": container with ID starting with 7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9 not found: ID does not exist" containerID="7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"
Sep 10 13:34:09.237823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.237787    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9"} err="failed to get container status \"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9\": rpc error: code = NotFound desc = could not find container \"7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9\": container with ID starting with 7cd74b66a4c1a5688ad10e3b9b3280fc486a4ba8810f21def0509b19307a71a9 not found: ID does not exist"
Sep 10 13:34:09.237823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.237802    2577 scope.go:117] "RemoveContainer" containerID="f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"
Sep 10 13:34:09.238033 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.238017    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7\": container with ID starting with f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7 not found: ID does not exist" containerID="f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"
Sep 10 13:34:09.238081 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238037    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7"} err="failed to get container status \"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7\": rpc error: code = NotFound desc = could not find container \"f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7\": container with ID starting with f8bc83209e2fad4f1e83f6e11900a718fcf5f2027f5c9c93305068cf55ef22e7 not found: ID does not exist"
Sep 10 13:34:09.238081 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238051    2577 scope.go:117] "RemoveContainer" containerID="b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"
Sep 10 13:34:09.238263 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.238247    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c\": container with ID starting with b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c not found: ID does not exist" containerID="b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"
Sep 10 13:34:09.238302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238266    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c"} err="failed to get container status \"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c\": rpc error: code = NotFound desc = could not find container \"b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c\": container with ID starting with b7329999a15ad43ecb7891e8f4e45a1e009088be2bbdcaade5d516f34709b63c not found: ID does not exist"
Sep 10 13:34:09.238302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238277    2577 scope.go:117] "RemoveContainer" containerID="b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"
Sep 10 13:34:09.238643 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.238488    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557\": container with ID starting with b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557 not found: ID does not exist" containerID="b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"
Sep 10 13:34:09.238643 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238532    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557"} err="failed to get container status \"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557\": rpc error: code = NotFound desc = could not find container \"b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557\": container with ID starting with b97749da26b181988b063c6855f181296eba6624429148951924f754d3203557 not found: ID does not exist"
Sep 10 13:34:09.238643 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238545    2577 scope.go:117] "RemoveContainer" containerID="b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"
Sep 10 13:34:09.238781 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.238766    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be\": container with ID starting with b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be not found: ID does not exist" containerID="b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"
Sep 10 13:34:09.238816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238786    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be"} err="failed to get container status \"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be\": rpc error: code = NotFound desc = could not find container \"b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be\": container with ID starting with b874b1299b96b57c8034feb2da5f7a0809d0e5f7970c10de013813199bd1f4be not found: ID does not exist"
Sep 10 13:34:09.238816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.238802    2577 scope.go:117] "RemoveContainer" containerID="98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f"
Sep 10 13:34:09.239045 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:09.239027    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f\": container with ID starting with 98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f not found: ID does not exist" containerID="98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f"
Sep 10 13:34:09.239087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:09.239048    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f"} err="failed to get container status \"98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f\": rpc error: code = NotFound desc = could not find container \"98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f\": container with ID starting with 98b04be6d233659e456471076fb4a8a15c44777783afd9b77f55126d1963110f not found: ID does not exist"
Sep 10 13:34:10.062469 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:10.062437    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" path="/var/lib/kubelet/pods/11d222c5-8048-420e-af3e-f18f0d1f630f/volumes"
Sep 10 13:34:11.538422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538381    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:11.538945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538805    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="prometheus" containerID="cri-o://513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac" gracePeriod=600
Sep 10 13:34:11.538945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538825    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-thanos" containerID="cri-o://06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54" gracePeriod=600
Sep 10 13:34:11.538945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538886    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-web" containerID="cri-o://69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e" gracePeriod=600
Sep 10 13:34:11.538945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538892    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="thanos-sidecar" containerID="cri-o://bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8" gracePeriod=600
Sep 10 13:34:11.538945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538891    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy" containerID="cri-o://cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164" gracePeriod=600
Sep 10 13:34:11.539187 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:11.538904    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="config-reloader" containerID="cri-o://d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612" gracePeriod=600
Sep 10 13:34:12.203634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203596    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54" exitCode=0
Sep 10 13:34:12.203634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203620    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164" exitCode=0
Sep 10 13:34:12.203634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203626    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8" exitCode=0
Sep 10 13:34:12.203634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203632    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612" exitCode=0
Sep 10 13:34:12.203634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203638    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac" exitCode=0
Sep 10 13:34:12.203958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"}
Sep 10 13:34:12.203958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"}
Sep 10 13:34:12.203958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"}
Sep 10 13:34:12.203958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"}
Sep 10 13:34:12.203958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.203803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"}
Sep 10 13:34:12.785331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.785296    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:12.800098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800131    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800226    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h94sd\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.800580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800644    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db\") pod \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\" (UID: \"fdf44d65-5eab-400a-b5be-0ffdc3c1d485\") "
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.801044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.800914    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.801365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.801333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.801906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.801878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.802024 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.801946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.802327 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.802258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.803077 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.803055    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.803253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.803188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.803653 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.803298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.803653 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.803598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out" (OuterVolumeSpecName: "config-out") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:34:12.803653 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.803625    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:34:12.804170 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.804122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:34:12.805440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.805413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config" (OuterVolumeSpecName: "config") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.805897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.805815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.805897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.805835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd" (OuterVolumeSpecName: "kube-api-access-h94sd") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "kube-api-access-h94sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:34:12.806061 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.805959    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.806122 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.806077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.806376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.806358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.806376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.806365    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:34:12.806666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.806637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.814951 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.814926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config" (OuterVolumeSpecName: "web-config") pod "fdf44d65-5eab-400a-b5be-0ffdc3c1d485" (UID: "fdf44d65-5eab-400a-b5be-0ffdc3c1d485"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902103    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902133    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-tls-assets\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902143    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902153    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902164    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902172    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h94sd\" (UniqueName: \"kubernetes.io/projected/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-kube-api-access-h94sd\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902181    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-web-config\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902188    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-metrics-client-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902196    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-metrics-client-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902205    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902214    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-db\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902221    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-grpc-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902230    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902238    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config-out\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902246    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902253    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-config\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902261    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902269    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:12.902590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:12.902277    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdf44d65-5eab-400a-b5be-0ffdc3c1d485-secret-kube-rbac-proxy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:34:13.210366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.210278    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerID="69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e" exitCode=0
Sep 10 13:34:13.210366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.210348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"}
Sep 10 13:34:13.210635 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.210368    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.210635 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.210421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fdf44d65-5eab-400a-b5be-0ffdc3c1d485","Type":"ContainerDied","Data":"ae1263a7301aa460b967fc0cc912f0037c036ae20248a16150892185d1741f8c"}
Sep 10 13:34:13.210635 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.210466    2577 scope.go:117] "RemoveContainer" containerID="06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"
Sep 10 13:34:13.218387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.218369    2577 scope.go:117] "RemoveContainer" containerID="cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"
Sep 10 13:34:13.225281 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.225263    2577 scope.go:117] "RemoveContainer" containerID="69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"
Sep 10 13:34:13.231456 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.231440    2577 scope.go:117] "RemoveContainer" containerID="bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"
Sep 10 13:34:13.236039 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.235482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:13.238750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.238733    2577 scope.go:117] "RemoveContainer" containerID="d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"
Sep 10 13:34:13.241903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.241881    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:13.245519 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.245493    2577 scope.go:117] "RemoveContainer" containerID="513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"
Sep 10 13:34:13.252742 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.252719    2577 scope.go:117] "RemoveContainer" containerID="aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3"
Sep 10 13:34:13.260955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.260918    2577 scope.go:117] "RemoveContainer" containerID="06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"
Sep 10 13:34:13.261404 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.261386    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54\": container with ID starting with 06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54 not found: ID does not exist" containerID="06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"
Sep 10 13:34:13.261479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.261413    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54"} err="failed to get container status \"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54\": rpc error: code = NotFound desc = could not find container \"06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54\": container with ID starting with 06ca28cacc3985058bb4cb3f4edad251c21ec50ddac3c7bad6b166dea303ea54 not found: ID does not exist"
Sep 10 13:34:13.261479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.261431    2577 scope.go:117] "RemoveContainer" containerID="cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"
Sep 10 13:34:13.261719 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.261702    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164\": container with ID starting with cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164 not found: ID does not exist" containerID="cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"
Sep 10 13:34:13.261761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.261724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164"} err="failed to get container status \"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164\": rpc error: code = NotFound desc = could not find container \"cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164\": container with ID starting with cb0bb7fc53aea4a6b87f4dbcba3a3c4c00973c0b114ff2071cd0620fb5f6f164 not found: ID does not exist"
Sep 10 13:34:13.261761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.261737    2577 scope.go:117] "RemoveContainer" containerID="69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"
Sep 10 13:34:13.261988 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.261967    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e\": container with ID starting with 69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e not found: ID does not exist" containerID="69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"
Sep 10 13:34:13.262032 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.261998    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e"} err="failed to get container status \"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e\": rpc error: code = NotFound desc = could not find container \"69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e\": container with ID starting with 69052efa9de74bfd7cdb88f78c99bb9401128a92484fcab47047717dcfa1d13e not found: ID does not exist"
Sep 10 13:34:13.262032 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262016    2577 scope.go:117] "RemoveContainer" containerID="bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"
Sep 10 13:34:13.262256 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.262239    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8\": container with ID starting with bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8 not found: ID does not exist" containerID="bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"
Sep 10 13:34:13.262296 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262263    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8"} err="failed to get container status \"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8\": rpc error: code = NotFound desc = could not find container \"bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8\": container with ID starting with bf7be44d0ee81fbfe96f104f4201301b9f195704784628a3fdcf1530fe0a3aa8 not found: ID does not exist"
Sep 10 13:34:13.262296 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262279    2577 scope.go:117] "RemoveContainer" containerID="d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"
Sep 10 13:34:13.262559 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.262540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612\": container with ID starting with d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612 not found: ID does not exist" containerID="d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"
Sep 10 13:34:13.262627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262561    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612"} err="failed to get container status \"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612\": rpc error: code = NotFound desc = could not find container \"d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612\": container with ID starting with d30993c2054e1aa6133f4619887f87eb4f25fdb18edf046f9656802dcf60d612 not found: ID does not exist"
Sep 10 13:34:13.262627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262574    2577 scope.go:117] "RemoveContainer" containerID="513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"
Sep 10 13:34:13.262825 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.262808    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac\": container with ID starting with 513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac not found: ID does not exist" containerID="513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"
Sep 10 13:34:13.262860 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262830    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac"} err="failed to get container status \"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac\": rpc error: code = NotFound desc = could not find container \"513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac\": container with ID starting with 513b8c181b536697015f147111253085e1bcc9c8e314543ac62ed990a32613ac not found: ID does not exist"
Sep 10 13:34:13.262860 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.262846    2577 scope.go:117] "RemoveContainer" containerID="aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3"
Sep 10 13:34:13.263091 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:34:13.263076    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3\": container with ID starting with aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3 not found: ID does not exist" containerID="aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3"
Sep 10 13:34:13.263127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.263103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3"} err="failed to get container status \"aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3\": rpc error: code = NotFound desc = could not find container \"aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3\": container with ID starting with aeb83980e9ace8594d414a869d873e4fe02b53a9bf1d8ab6d69eb9a34690c5e3 not found: ID does not exist"
Sep 10 13:34:13.276176 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276149    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:13.276436 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276421    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="prometheus"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="prometheus"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276452    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="prom-label-proxy"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="prom-label-proxy"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276477    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276503    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="init-config-reloader"
Sep 10 13:34:13.276531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276528    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="init-config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276539    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276556    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="alertmanager"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276565    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="alertmanager"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276576    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276585    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276596    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276613    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-metric"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276621    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-metric"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276629    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276637    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="init-config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276659    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="init-config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276670    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276678    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276687    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-thanos"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276694    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-thanos"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276705    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="thanos-sidecar"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276713    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="thanos-sidecar"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276806    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276816    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="prom-label-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276826    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="config-reloader"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276833    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-thanos"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276843    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-metric"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276852    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276861    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276868    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="thanos-sidecar"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276877    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="kube-rbac-proxy"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276884    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="alertmanager"
Sep 10 13:34:13.276880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276892    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d222c5-8048-420e-af3e-f18f0d1f630f" containerName="kube-rbac-proxy-web"
Sep 10 13:34:13.278288 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.276900    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" containerName="prometheus"
Sep 10 13:34:13.282340 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.282319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.286013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.285989    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b01javqsmojgo\""
Sep 10 13:34:13.286675 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.286658    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 13:34:13.286997 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.286980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 13:34:13.287194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287180    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 13:34:13.287388 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287357    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 13:34:13.287479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287389    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmdnc\""
Sep 10 13:34:13.287581 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 13:34:13.287700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 13:34:13.287779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 13:34:13.287924 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.287884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 13:34:13.288088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.288009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 13:34:13.288088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.288027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 13:34:13.288210 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.288098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 13:34:13.288287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.288272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 13:34:13.288808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.288784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 13:34:13.293557 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.293540    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 13:34:13.298587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.298384    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 13:34:13.299990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.299972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:13.305714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfl46\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-kube-api-access-jfl46\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.305818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.305982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.306770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.306155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfl46\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-kube-api-access-jfl46\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.407835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.408567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.408567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.408567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.408567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.407999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.408567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.408413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.409223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.408655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.409223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.409137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.410605 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.410147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.410791 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.410762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.411342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.411314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.411779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.411751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.411882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.411779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.412040 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.412018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config-out\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.412202 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.412178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.412268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.412244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.412502 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.412480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.413042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.413027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-web-config\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.413250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.413225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.413547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.413532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.413924 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.413909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.422663 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.422638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfl46\" (UniqueName: \"kubernetes.io/projected/ed9b0e77-0378-404d-ac0a-9eb6c5ff92af-kube-api-access-jfl46\") pod \"prometheus-k8s-0\" (UID: \"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.592168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.592073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:34:13.879206 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:13.879177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 13:34:13.880957 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:34:13.880933    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9b0e77_0378_404d_ac0a_9eb6c5ff92af.slice/crio-9315153c6ac04d6316263d2fe73fafba5927d19fd10cde81041882babc40e107 WatchSource:0}: Error finding container 9315153c6ac04d6316263d2fe73fafba5927d19fd10cde81041882babc40e107: Status 404 returned error can't find the container with id 9315153c6ac04d6316263d2fe73fafba5927d19fd10cde81041882babc40e107
Sep 10 13:34:14.061710 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:14.061670    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf44d65-5eab-400a-b5be-0ffdc3c1d485" path="/var/lib/kubelet/pods/fdf44d65-5eab-400a-b5be-0ffdc3c1d485/volumes"
Sep 10 13:34:14.215268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:14.215180    2577 generic.go:356] "Generic (PLEG): container finished" podID="ed9b0e77-0378-404d-ac0a-9eb6c5ff92af" containerID="5475d05c105149eb4e94f4bdf26eb88e0f71f781f34322d74710ed4a603cd3fd" exitCode=0
Sep 10 13:34:14.215422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:14.215269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerDied","Data":"5475d05c105149eb4e94f4bdf26eb88e0f71f781f34322d74710ed4a603cd3fd"}
Sep 10 13:34:14.215422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:14.215343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"9315153c6ac04d6316263d2fe73fafba5927d19fd10cde81041882babc40e107"}
Sep 10 13:34:15.223703 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"3d2d49d162af513bd9709971e521572a95699ee8152771070ea45a258df917f5"}
Sep 10 13:34:15.224091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"8e371884762ed45a58798275614d2aca220cd3d592bc3cc80bac2ed4f13008b8"}
Sep 10 13:34:15.224091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"ea59c0cf3544c89bc5b4aa062f805680b09b3e98098d7b3fdb94f7ba7685a372"}
Sep 10 13:34:15.224091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"b550e749e2c3185b546d6d5f121b5a538cbccabb8c3176f576efe6dac7953f14"}
Sep 10 13:34:15.224091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"6926c878c5031df6dedef0d0da6407c42813a86f60f4c36fa30f5e0cabcca4e0"}
Sep 10 13:34:15.224091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.223899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ed9b0e77-0378-404d-ac0a-9eb6c5ff92af","Type":"ContainerStarted","Data":"54efb841f4d1a38e119aba7a5e8c59f7e840a824dfa5fdf608105da3dd580e41"}
Sep 10 13:34:15.256981 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:15.256925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.235707977 podStartE2EDuration="2.256900438s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.192461ms" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:15.254802452 +0000 UTC m=+339.709677651" watchObservedRunningTime="2026-09-10 13:34:15.256900438 +0000 UTC m=+339.711775601"
Sep 10 13:34:18.592723 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:34:18.592682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:35:13.592866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:13.592825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:35:13.607702 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:13.607673    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:35:14.451936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:14.451907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 13:35:35.988972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:35.988943    2577 scope.go:117] "RemoveContainer" containerID="f49896dd033690519683cc794fc1eed96b1bc3a4f8c5383d060cd6a6a93cd0a6"
Sep 10 13:35:35.996220 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:35.996202    2577 scope.go:117] "RemoveContainer" containerID="3f6e93a522a8cd96b17b551038175c0fa17329388d3ac666585131e040104d1b"
Sep 10 13:35:36.003078 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.003060    2577 scope.go:117] "RemoveContainer" containerID="6f538d4c550f473b58cc99c784a99a9b56aea59236b8ebda96903354c7b7f888"
Sep 10 13:35:36.009126 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.009107    2577 scope.go:117] "RemoveContainer" containerID="a81e8856e6c3ba5a027d61ed43b655f1c876ef456ff6a851f180767f9c56075c"
Sep 10 13:35:36.015317 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.015298    2577 scope.go:117] "RemoveContainer" containerID="5bf9c428b5a512f68e6aa4b40c9a55067b23650b3f4b5d515e37279510f6d97f"
Sep 10 13:35:36.022843 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.022824    2577 scope.go:117] "RemoveContainer" containerID="de8b36c8b8be0bbd31f87430f76064292ef9269777f5e7d2bec7705836477699"
Sep 10 13:35:36.029593 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.029576    2577 scope.go:117] "RemoveContainer" containerID="db4b1d9585f3bdc1efb47676adc1aa6ec5aab9e01a71a09655cc224416cf40b6"
Sep 10 13:35:36.036666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:35:36.036646    2577 scope.go:117] "RemoveContainer" containerID="fde8a46745f17bd625ab32efbf0e629ee4ce176770fb9e4a5302afda6f9f4a4f"
Sep 10 13:37:58.826064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.826026    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"]
Sep 10 13:37:58.831090 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.831073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.835390 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.835368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 13:37:58.835571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.835461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 13:37:58.835955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.835939    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6c66s\""
Sep 10 13:37:58.836454 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.836436    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 13:37:58.836568 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.836525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 13:37:58.854959 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.854935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"]
Sep 10 13:37:58.887365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.887337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.887539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.887377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.887539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.887398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhbz\" (UniqueName: \"kubernetes.io/projected/9a6382dc-365b-46b3-9a9b-1a8a0a342154-kube-api-access-cnhbz\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.887539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.887446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.887539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.887487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhbz\" (UniqueName: \"kubernetes.io/projected/9a6382dc-365b-46b3-9a9b-1a8a0a342154-kube-api-access-cnhbz\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.988727 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.988705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-manifests\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.990610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.990576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a6382dc-365b-46b3-9a9b-1a8a0a342154-tmp\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.990846 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.990828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-webhook-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.990952 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.990933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a6382dc-365b-46b3-9a9b-1a8a0a342154-apiservice-cert\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:58.997878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:58.997853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhbz\" (UniqueName: \"kubernetes.io/projected/9a6382dc-365b-46b3-9a9b-1a8a0a342154-kube-api-access-cnhbz\") pod \"opendatahub-operator-controller-manager-579d56558b-kzx4f\" (UID: \"9a6382dc-365b-46b3-9a9b-1a8a0a342154\") " pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:59.141203 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:59.141173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:37:59.450948 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:59.450718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"]
Sep 10 13:37:59.453947 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:37:59.453921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6382dc_365b_46b3_9a9b_1a8a0a342154.slice/crio-f5344b697295053ef5e2a9a8489aac88689146b7a0d072f6b7e293400128f42b WatchSource:0}: Error finding container f5344b697295053ef5e2a9a8489aac88689146b7a0d072f6b7e293400128f42b: Status 404 returned error can't find the container with id f5344b697295053ef5e2a9a8489aac88689146b7a0d072f6b7e293400128f42b
Sep 10 13:37:59.455685 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:59.455668    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:37:59.998072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:37:59.998041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f" event={"ID":"9a6382dc-365b-46b3-9a9b-1a8a0a342154","Type":"ContainerStarted","Data":"f5344b697295053ef5e2a9a8489aac88689146b7a0d072f6b7e293400128f42b"}
Sep 10 13:38:03.011758 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:03.011720    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f" event={"ID":"9a6382dc-365b-46b3-9a9b-1a8a0a342154","Type":"ContainerStarted","Data":"39b8ee20af8e965ff85b6a3d374049e7219ef61d7c1c88a397c44162eb36d3cb"}
Sep 10 13:38:04.016947 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:04.016911    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a6382dc-365b-46b3-9a9b-1a8a0a342154" containerID="39b8ee20af8e965ff85b6a3d374049e7219ef61d7c1c88a397c44162eb36d3cb" exitCode=0
Sep 10 13:38:04.017344 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:04.017000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f" event={"ID":"9a6382dc-365b-46b3-9a9b-1a8a0a342154","Type":"ContainerDied","Data":"39b8ee20af8e965ff85b6a3d374049e7219ef61d7c1c88a397c44162eb36d3cb"}
Sep 10 13:38:05.024225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:05.024141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f" event={"ID":"9a6382dc-365b-46b3-9a9b-1a8a0a342154","Type":"ContainerStarted","Data":"8e2a1db13dd1d334b02ef7dbde2bca258f67c16782683dc17e97abee6996fd8e"}
Sep 10 13:38:05.024605 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:05.024269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:38:05.050432 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:05.050372    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f" podStartSLOduration=2.422307161 podStartE2EDuration="7.050357104s" totalImagesPullingTime="3.615043601s" totalInitContainerRuntime="1.013006342s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:37:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:05.049928761 +0000 UTC m=+569.504803925" watchObservedRunningTime="2026-09-10 13:38:05.050357104 +0000 UTC m=+569.505232266"
Sep 10 13:38:16.030576 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:16.030543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-579d56558b-kzx4f"
Sep 10 13:38:18.909766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.909729    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"]
Sep 10 13:38:18.919440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.919414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:18.921832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.921809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 13:38:18.922800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.922780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 13:38:18.922892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.922854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9grd5\""
Sep 10 13:38:18.923592 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.923571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 13:38:18.923691 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.923662    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 13:38:18.928827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:18.928802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"]
Sep 10 13:38:19.055925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.055866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853e7098-e0ef-4923-a8a9-0cbd513d423d-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.056108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.055999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gld2\" (UniqueName: \"kubernetes.io/projected/853e7098-e0ef-4923-a8a9-0cbd513d423d-kube-api-access-4gld2\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.056108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.056050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853e7098-e0ef-4923-a8a9-0cbd513d423d-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.157168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.157114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gld2\" (UniqueName: \"kubernetes.io/projected/853e7098-e0ef-4923-a8a9-0cbd513d423d-kube-api-access-4gld2\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.157168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.157168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853e7098-e0ef-4923-a8a9-0cbd513d423d-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.157455 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.157199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853e7098-e0ef-4923-a8a9-0cbd513d423d-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.159441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.159417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/853e7098-e0ef-4923-a8a9-0cbd513d423d-tmp\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.159689 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.159666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/853e7098-e0ef-4923-a8a9-0cbd513d423d-tls-certs\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.166138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.166081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gld2\" (UniqueName: \"kubernetes.io/projected/853e7098-e0ef-4923-a8a9-0cbd513d423d-kube-api-access-4gld2\") pod \"kube-auth-proxy-69ddc5c8c5-jr7wq\" (UID: \"853e7098-e0ef-4923-a8a9-0cbd513d423d\") " pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.230117 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.230076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"
Sep 10 13:38:19.506490 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:19.506462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq"]
Sep 10 13:38:19.508887 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:38:19.508848    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853e7098_e0ef_4923_a8a9_0cbd513d423d.slice/crio-49cd323ca52289e877079c067a0507bcd1060b7060e3b81b6f1220b3d458ec58 WatchSource:0}: Error finding container 49cd323ca52289e877079c067a0507bcd1060b7060e3b81b6f1220b3d458ec58: Status 404 returned error can't find the container with id 49cd323ca52289e877079c067a0507bcd1060b7060e3b81b6f1220b3d458ec58
Sep 10 13:38:20.076208 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.076176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq" event={"ID":"853e7098-e0ef-4923-a8a9-0cbd513d423d","Type":"ContainerStarted","Data":"49cd323ca52289e877079c067a0507bcd1060b7060e3b81b6f1220b3d458ec58"}
Sep 10 13:38:20.193566 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.193481    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:20.195761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.195736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.198407 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.198380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 13:38:20.198407 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.198400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 13:38:20.198654 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.198452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 13:38:20.198654 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.198381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 13:38:20.198851 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.198832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7xz5c\""
Sep 10 13:38:20.214395 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.214370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:20.266722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx64v\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.266988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.266895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367500 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx64v\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.367943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.367765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.368730 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.368705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.371027 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.370992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.371213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.371192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.372213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.372171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.372469 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.372444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.377613 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.377587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx64v\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.380339 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.379763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtpkc\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.507449 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.507401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:20.813278 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:20.813251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:20.815501 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:38:20.815469    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a759334_926b_4b30_a559_89009e101ee9.slice/crio-faff0c1abbb59c1a3be52871b1196bc8517d88991a92e6d91df63ae5c3dca17a WatchSource:0}: Error finding container faff0c1abbb59c1a3be52871b1196bc8517d88991a92e6d91df63ae5c3dca17a: Status 404 returned error can't find the container with id faff0c1abbb59c1a3be52871b1196bc8517d88991a92e6d91df63ae5c3dca17a
Sep 10 13:38:21.081841 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:21.081763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" event={"ID":"7a759334-926b-4b30-a559-89009e101ee9","Type":"ContainerStarted","Data":"faff0c1abbb59c1a3be52871b1196bc8517d88991a92e6d91df63ae5c3dca17a"}
Sep 10 13:38:23.092612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:23.092569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq" event={"ID":"853e7098-e0ef-4923-a8a9-0cbd513d423d","Type":"ContainerStarted","Data":"aaa1693838186896f22c96b8bfd8c5992fd5bb471958086eaba4a98d5a2dbdc6"}
Sep 10 13:38:23.119880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:23.119819    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69ddc5c8c5-jr7wq" podStartSLOduration=2.290605583 podStartE2EDuration="5.11979664s" totalImagesPullingTime="2.829191057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:23.119702783 +0000 UTC m=+587.574577974" watchObservedRunningTime="2026-09-10 13:38:23.11979664 +0000 UTC m=+587.574671819"
Sep 10 13:38:24.187577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:24.187534    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:38:24.187861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:24.187611    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:38:25.101857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:25.101814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" event={"ID":"7a759334-926b-4b30-a559-89009e101ee9","Type":"ContainerStarted","Data":"5f4020002336c4c5d8d78b694e3e6690b9055dfe459355dcfe71ec94650ae01b"}
Sep 10 13:38:25.102115 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:25.101975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:25.103752 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:25.103727    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vtpkc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:38:25.103862 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:25.103772    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" podUID="7a759334-926b-4b30-a559-89009e101ee9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:38:25.151875 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:25.151819    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" podStartSLOduration=1.782762459 podStartE2EDuration="5.151802932s" totalImagesPullingTime="3.369040473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:25.150721437 +0000 UTC m=+589.605596603" watchObservedRunningTime="2026-09-10 13:38:25.151802932 +0000 UTC m=+589.606678097"
Sep 10 13:38:26.106391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:26.106360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:36.003866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:36.003841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:38:36.004482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:36.004458    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:38:42.960392 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:42.960354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:42.962803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:42.960549    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" podUID="7a759334-926b-4b30-a559-89009e101ee9" containerName="discovery" containerID="cri-o://5f4020002336c4c5d8d78b694e3e6690b9055dfe459355dcfe71ec94650ae01b" gracePeriod=30
Sep 10 13:38:43.170455 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.170419    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a759334-926b-4b30-a559-89009e101ee9" containerID="5f4020002336c4c5d8d78b694e3e6690b9055dfe459355dcfe71ec94650ae01b" exitCode=0
Sep 10 13:38:43.170652 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.170492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" event={"ID":"7a759334-926b-4b30-a559-89009e101ee9","Type":"ContainerDied","Data":"5f4020002336c4c5d8d78b694e3e6690b9055dfe459355dcfe71ec94650ae01b"}
Sep 10 13:38:43.195502 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.195469    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:43.269007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.268923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.268975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269000    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx64v\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig\") pod \"7a759334-926b-4b30-a559-89009e101ee9\" (UID: \"7a759334-926b-4b30-a559-89009e101ee9\") "
Sep 10 13:38:43.269590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.269559    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:38:43.271426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:43.271589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:43.271589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271548    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs" (OuterVolumeSpecName: "local-certs") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:43.271589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271555    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts" (OuterVolumeSpecName: "cacerts") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:43.271757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:43.271757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.271703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v" (OuterVolumeSpecName: "kube-api-access-bx64v") pod "7a759334-926b-4b30-a559-89009e101ee9" (UID: "7a759334-926b-4b30-a559-89009e101ee9"). InnerVolumeSpecName "kube-api-access-bx64v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:43.369728 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369694    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-ca-configmap\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369728 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369726    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-csr-dns-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369737    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx64v\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-kube-api-access-bx64v\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369749    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a759334-926b-4b30-a559-89009e101ee9-local-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369760    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a759334-926b-4b30-a559-89009e101ee9-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369771    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-istio-kubeconfig\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:43.369935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:43.369782    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a759334-926b-4b30-a559-89009e101ee9-cacerts\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:38:44.175247 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.175213    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"
Sep 10 13:38:44.175690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.175212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc" event={"ID":"7a759334-926b-4b30-a559-89009e101ee9","Type":"ContainerDied","Data":"faff0c1abbb59c1a3be52871b1196bc8517d88991a92e6d91df63ae5c3dca17a"}
Sep 10 13:38:44.175690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.175378    2577 scope.go:117] "RemoveContainer" containerID="5f4020002336c4c5d8d78b694e3e6690b9055dfe459355dcfe71ec94650ae01b"
Sep 10 13:38:44.228292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.228255    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:44.240665 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.240633    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtpkc"]
Sep 10 13:38:44.387953 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.387916    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-722tv"]
Sep 10 13:38:44.388280 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.388263    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a759334-926b-4b30-a559-89009e101ee9" containerName="discovery"
Sep 10 13:38:44.388365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.388283    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a759334-926b-4b30-a559-89009e101ee9" containerName="discovery"
Sep 10 13:38:44.388365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.388357    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a759334-926b-4b30-a559-89009e101ee9" containerName="discovery"
Sep 10 13:38:44.392456 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.392434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.398641 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.398615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-q7ttn\""
Sep 10 13:38:44.398765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.398645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 13:38:44.411220 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.411192    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"]
Sep 10 13:38:44.414178 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.414160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.417970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.417950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5wg7s\""
Sep 10 13:38:44.420910 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.420889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 13:38:44.466274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.466242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-722tv"]
Sep 10 13:38:44.476949 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.476921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l988w\" (UniqueName: \"kubernetes.io/projected/f77aec85-e054-4d1e-a67a-a946072310ec-kube-api-access-l988w\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.477113 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.476967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kqgt\" (UniqueName: \"kubernetes.io/projected/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-kube-api-access-5kqgt\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.477113 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.477018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-cert\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.477113 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.477082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f77aec85-e054-4d1e-a67a-a946072310ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.578109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.578072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-cert\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.578313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.578134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f77aec85-e054-4d1e-a67a-a946072310ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.578313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.578213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l988w\" (UniqueName: \"kubernetes.io/projected/f77aec85-e054-4d1e-a67a-a946072310ec-kube-api-access-l988w\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.578313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.578259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kqgt\" (UniqueName: \"kubernetes.io/projected/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-kube-api-access-5kqgt\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.580809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.580781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f77aec85-e054-4d1e-a67a-a946072310ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.581075 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.581055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-cert\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.594561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.594485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kqgt\" (UniqueName: \"kubernetes.io/projected/ca23f6d2-928f-412e-8e52-b5f9be9a3f85-kube-api-access-5kqgt\") pod \"kserve-controller-manager-d6f4b65b8-722tv\" (UID: \"ca23f6d2-928f-412e-8e52-b5f9be9a3f85\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.597224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.597184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"]
Sep 10 13:38:44.597525 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.597482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l988w\" (UniqueName: \"kubernetes.io/projected/f77aec85-e054-4d1e-a67a-a946072310ec-kube-api-access-l988w\") pod \"llmisvc-controller-manager-777c6bd7d5-wr98b\" (UID: \"f77aec85-e054-4d1e-a67a-a946072310ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:44.702681 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.702577    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:44.723984 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:44.723955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:45.051799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:45.051672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-722tv"]
Sep 10 13:38:45.056381 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:38:45.056351    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca23f6d2_928f_412e_8e52_b5f9be9a3f85.slice/crio-351269f0dd20e528f866a972df5399de2ee19e4301f60e279189c0c00b941ed4 WatchSource:0}: Error finding container 351269f0dd20e528f866a972df5399de2ee19e4301f60e279189c0c00b941ed4: Status 404 returned error can't find the container with id 351269f0dd20e528f866a972df5399de2ee19e4301f60e279189c0c00b941ed4
Sep 10 13:38:45.088650 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:45.088616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"]
Sep 10 13:38:45.092456 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:38:45.092428    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf77aec85_e054_4d1e_a67a_a946072310ec.slice/crio-2ca65077473e4a3545dfa7c830735ff6f3cd545674515151c1b621cd647118c0 WatchSource:0}: Error finding container 2ca65077473e4a3545dfa7c830735ff6f3cd545674515151c1b621cd647118c0: Status 404 returned error can't find the container with id 2ca65077473e4a3545dfa7c830735ff6f3cd545674515151c1b621cd647118c0
Sep 10 13:38:45.180356 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:45.180320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b" event={"ID":"f77aec85-e054-4d1e-a67a-a946072310ec","Type":"ContainerStarted","Data":"2ca65077473e4a3545dfa7c830735ff6f3cd545674515151c1b621cd647118c0"}
Sep 10 13:38:45.181405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:45.181383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv" event={"ID":"ca23f6d2-928f-412e-8e52-b5f9be9a3f85","Type":"ContainerStarted","Data":"351269f0dd20e528f866a972df5399de2ee19e4301f60e279189c0c00b941ed4"}
Sep 10 13:38:46.064807 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:46.064770    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a759334-926b-4b30-a559-89009e101ee9" path="/var/lib/kubelet/pods/7a759334-926b-4b30-a559-89009e101ee9/volumes"
Sep 10 13:38:48.195839 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.195732    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b" event={"ID":"f77aec85-e054-4d1e-a67a-a946072310ec","Type":"ContainerStarted","Data":"56b307dee1ccb6ec0cd6f9ba4acc8246ab4a99d30d2d2f22610821adc9626889"}
Sep 10 13:38:48.195839 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.195804    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:38:48.197298 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.197274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv" event={"ID":"ca23f6d2-928f-412e-8e52-b5f9be9a3f85","Type":"ContainerStarted","Data":"b4f91d8ce6b7d25093e5e0b6f170ec4f441c9301b55b6975bd65fdb835e1c461"}
Sep 10 13:38:48.197416 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.197406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:38:48.225313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.225261    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b" podStartSLOduration=1.342693792 podStartE2EDuration="4.225245301s" totalImagesPullingTime="2.882551509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:48.224580178 +0000 UTC m=+612.679455364" watchObservedRunningTime="2026-09-10 13:38:48.225245301 +0000 UTC m=+612.680120463"
Sep 10 13:38:48.264230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:38:48.264166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv" podStartSLOduration=1.388730175 podStartE2EDuration="4.264143242s" totalImagesPullingTime="2.875413067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:38:48.263562969 +0000 UTC m=+612.718438132" watchObservedRunningTime="2026-09-10 13:38:48.264143242 +0000 UTC m=+612.719018408"
Sep 10 13:39:19.203063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:19.202988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wr98b"
Sep 10 13:39:19.206002 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:19.205980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-722tv"
Sep 10 13:39:48.430090 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.430053    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"]
Sep 10 13:39:48.434132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.434112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.436934 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.436908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 13:39:48.437048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.436939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 13:39:48.437048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.436955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lsk9v\""
Sep 10 13:39:48.450283 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.450256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"]
Sep 10 13:39:48.491200 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.491168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g549\" (UniqueName: \"kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.491200 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.491205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.592243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.592206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g549\" (UniqueName: \"kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.592243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.592248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.592761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.592728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.611067 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.611041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g549\" (UniqueName: \"kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:48.743966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:48.743875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:49.047536 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:49.047492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"]
Sep 10 13:39:49.050372 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:39:49.050341    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9065e732_f18e_4355_bb75_2fe20d8dd6a6.slice/crio-cf601434a0d9b84f43dd21c669dcc81ed03e2f4471a7a5dc773c4dff00bc8311 WatchSource:0}: Error finding container cf601434a0d9b84f43dd21c669dcc81ed03e2f4471a7a5dc773c4dff00bc8311: Status 404 returned error can't find the container with id cf601434a0d9b84f43dd21c669dcc81ed03e2f4471a7a5dc773c4dff00bc8311
Sep 10 13:39:49.424152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:49.424113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds" event={"ID":"9065e732-f18e-4355-bb75-2fe20d8dd6a6","Type":"ContainerStarted","Data":"cf601434a0d9b84f43dd21c669dcc81ed03e2f4471a7a5dc773c4dff00bc8311"}
Sep 10 13:39:51.035916 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.035875    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-n7mn9"]
Sep 10 13:39:51.044823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.044779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:51.047646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.047623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-m7bsb\""
Sep 10 13:39:51.056337 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.056300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-n7mn9"]
Sep 10 13:39:51.117239 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.117183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcmn\" (UniqueName: \"kubernetes.io/projected/f086ce38-0404-4b44-916e-48515aa6c593-kube-api-access-mjcmn\") pod \"authorino-operator-76b478bd78-n7mn9\" (UID: \"f086ce38-0404-4b44-916e-48515aa6c593\") " pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:51.217850 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.217806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcmn\" (UniqueName: \"kubernetes.io/projected/f086ce38-0404-4b44-916e-48515aa6c593-kube-api-access-mjcmn\") pod \"authorino-operator-76b478bd78-n7mn9\" (UID: \"f086ce38-0404-4b44-916e-48515aa6c593\") " pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:51.233778 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.233746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcmn\" (UniqueName: \"kubernetes.io/projected/f086ce38-0404-4b44-916e-48515aa6c593-kube-api-access-mjcmn\") pod \"authorino-operator-76b478bd78-n7mn9\" (UID: \"f086ce38-0404-4b44-916e-48515aa6c593\") " pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:51.359716 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.359638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:51.776595 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:51.776562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-n7mn9"]
Sep 10 13:39:51.780439 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:39:51.780404    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf086ce38_0404_4b44_916e_48515aa6c593.slice/crio-19d14a8fb6845e71f0267072cff22111d3c1272d6c43a25a6dae144e09726b63 WatchSource:0}: Error finding container 19d14a8fb6845e71f0267072cff22111d3c1272d6c43a25a6dae144e09726b63: Status 404 returned error can't find the container with id 19d14a8fb6845e71f0267072cff22111d3c1272d6c43a25a6dae144e09726b63
Sep 10 13:39:52.441409 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:52.441374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9" event={"ID":"f086ce38-0404-4b44-916e-48515aa6c593","Type":"ContainerStarted","Data":"19d14a8fb6845e71f0267072cff22111d3c1272d6c43a25a6dae144e09726b63"}
Sep 10 13:39:53.201108 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.201046    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"]
Sep 10 13:39:53.204268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.204253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:53.206606 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.206582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kdpq8\""
Sep 10 13:39:53.217313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.217291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"]
Sep 10 13:39:53.236714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.236682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2f2\" (UniqueName: \"kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2\") pod \"limitador-operator-controller-manager-74495c69f8-dg27h\" (UID: \"84d634a5-361c-4627-ac67-88580453fc92\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:53.337323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.337282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2f2\" (UniqueName: \"kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2\") pod \"limitador-operator-controller-manager-74495c69f8-dg27h\" (UID: \"84d634a5-361c-4627-ac67-88580453fc92\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:53.349647 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.349570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2f2\" (UniqueName: \"kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2\") pod \"limitador-operator-controller-manager-74495c69f8-dg27h\" (UID: \"84d634a5-361c-4627-ac67-88580453fc92\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:53.449532 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.449471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds" event={"ID":"9065e732-f18e-4355-bb75-2fe20d8dd6a6","Type":"ContainerStarted","Data":"ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e"}
Sep 10 13:39:53.449966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.449688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:39:53.490427 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.490370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds" podStartSLOduration=1.451225944 podStartE2EDuration="5.490353222s" totalImagesPullingTime="4.039127278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:53.487052171 +0000 UTC m=+677.941927333" watchObservedRunningTime="2026-09-10 13:39:53.490353222 +0000 UTC m=+677.945228384"
Sep 10 13:39:53.513635 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.513603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:53.832033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:53.831951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"]
Sep 10 13:39:53.948816 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:39:53.948778    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d634a5_361c_4627_ac67_88580453fc92.slice/crio-b7754a241968a7a33af9435ecf827e24856299cd77caff8cb83359db7cf5203f WatchSource:0}: Error finding container b7754a241968a7a33af9435ecf827e24856299cd77caff8cb83359db7cf5203f: Status 404 returned error can't find the container with id b7754a241968a7a33af9435ecf827e24856299cd77caff8cb83359db7cf5203f
Sep 10 13:39:54.455154 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:54.455121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h" event={"ID":"84d634a5-361c-4627-ac67-88580453fc92","Type":"ContainerStarted","Data":"b7754a241968a7a33af9435ecf827e24856299cd77caff8cb83359db7cf5203f"}
Sep 10 13:39:54.456690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:54.456662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9" event={"ID":"f086ce38-0404-4b44-916e-48515aa6c593","Type":"ContainerStarted","Data":"3e91214071d19adc6246864db77e0af1556e3254b0698a57126b22fe0c2ccc8d"}
Sep 10 13:39:54.456810 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:54.456749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:39:54.493688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:54.493640    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9" podStartSLOduration=1.272574721 podStartE2EDuration="3.493623783s" totalImagesPullingTime="2.221049062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:54.490436243 +0000 UTC m=+678.945311408" watchObservedRunningTime="2026-09-10 13:39:54.493623783 +0000 UTC m=+678.948498946"
Sep 10 13:39:55.535132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.535110    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"]
Sep 10 13:39:55.539015 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.538984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:55.541384 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.541354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 13:39:55.541522 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.541409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-58zvk\""
Sep 10 13:39:55.555324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.555293    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"]
Sep 10 13:39:55.661802 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.661769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9885k\" (UniqueName: \"kubernetes.io/projected/acc559a0-90cf-4547-840c-096c9d24dafb-kube-api-access-9885k\") pod \"dns-operator-controller-manager-7b84cd7c9c-2blzk\" (UID: \"acc559a0-90cf-4547-840c-096c9d24dafb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:55.762970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.762885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9885k\" (UniqueName: \"kubernetes.io/projected/acc559a0-90cf-4547-840c-096c9d24dafb-kube-api-access-9885k\") pod \"dns-operator-controller-manager-7b84cd7c9c-2blzk\" (UID: \"acc559a0-90cf-4547-840c-096c9d24dafb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:55.774732 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.774707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9885k\" (UniqueName: \"kubernetes.io/projected/acc559a0-90cf-4547-840c-096c9d24dafb-kube-api-access-9885k\") pod \"dns-operator-controller-manager-7b84cd7c9c-2blzk\" (UID: \"acc559a0-90cf-4547-840c-096c9d24dafb\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:55.849572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:55.849538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:56.130823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.130799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"]
Sep 10 13:39:56.132405 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:39:56.132379    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc559a0_90cf_4547_840c_096c9d24dafb.slice/crio-6a6c852cc2099f9fe93d48948690ed4f835cd841e0db98bf8731f47447ec8e7a WatchSource:0}: Error finding container 6a6c852cc2099f9fe93d48948690ed4f835cd841e0db98bf8731f47447ec8e7a: Status 404 returned error can't find the container with id 6a6c852cc2099f9fe93d48948690ed4f835cd841e0db98bf8731f47447ec8e7a
Sep 10 13:39:56.307584 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.307544    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"]
Sep 10 13:39:56.312249 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.312223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.314775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.314754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 13:39:56.314926 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.314836    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b9txb\""
Sep 10 13:39:56.315103 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.315087    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 13:39:56.324559 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.324533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"]
Sep 10 13:39:56.368619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.368583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlxs\" (UniqueName: \"kubernetes.io/projected/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-kube-api-access-tzlxs\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.368816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.368661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.368816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.368682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.469142 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.469142 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.469142 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlxs\" (UniqueName: \"kubernetes.io/projected/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-kube-api-access-tzlxs\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.469435 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:39:56.469185    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 13:39:56.469435 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:39:56.469250    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert podName:7f68e2ce-7a1d-47df-8e65-db2edeef86e8 nodeName:}" failed. No retries permitted until 2026-09-10 13:39:56.969229708 +0000 UTC m=+681.424104872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-t7glb" (UID: "7f68e2ce-7a1d-47df-8e65-db2edeef86e8") : secret "plugin-serving-cert" not found
Sep 10 13:39:56.469570 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h" event={"ID":"84d634a5-361c-4627-ac67-88580453fc92","Type":"ContainerStarted","Data":"ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c"}
Sep 10 13:39:56.469656 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:39:56.469857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.469837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.470863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.470844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk" event={"ID":"acc559a0-90cf-4547-840c-096c9d24dafb","Type":"ContainerStarted","Data":"6a6c852cc2099f9fe93d48948690ed4f835cd841e0db98bf8731f47447ec8e7a"}
Sep 10 13:39:56.482026 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.481996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlxs\" (UniqueName: \"kubernetes.io/projected/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-kube-api-access-tzlxs\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.498893 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.498850    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h" podStartSLOduration=1.9827900120000002 podStartE2EDuration="3.498834867s" totalImagesPullingTime="1.516044855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:56.49663551 +0000 UTC m=+680.951510673" watchObservedRunningTime="2026-09-10 13:39:56.498834867 +0000 UTC m=+680.953710030"
Sep 10 13:39:56.974275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.974229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:56.976771 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:56.976745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f68e2ce-7a1d-47df-8e65-db2edeef86e8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-t7glb\" (UID: \"7f68e2ce-7a1d-47df-8e65-db2edeef86e8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:57.222023 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:57.221978    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"
Sep 10 13:39:57.507063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:57.506976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb"]
Sep 10 13:39:57.510087 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:39:57.510053    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f68e2ce_7a1d_47df_8e65_db2edeef86e8.slice/crio-bdfc7a8219cf97bb003ea7f559477b18c43bc6a466f1d16a6f78ee7c520b021b WatchSource:0}: Error finding container bdfc7a8219cf97bb003ea7f559477b18c43bc6a466f1d16a6f78ee7c520b021b: Status 404 returned error can't find the container with id bdfc7a8219cf97bb003ea7f559477b18c43bc6a466f1d16a6f78ee7c520b021b
Sep 10 13:39:58.488138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:58.488094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb" event={"ID":"7f68e2ce-7a1d-47df-8e65-db2edeef86e8","Type":"ContainerStarted","Data":"bdfc7a8219cf97bb003ea7f559477b18c43bc6a466f1d16a6f78ee7c520b021b"}
Sep 10 13:39:58.492026 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:58.491986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk" event={"ID":"acc559a0-90cf-4547-840c-096c9d24dafb","Type":"ContainerStarted","Data":"53343d85b08d322f83181999e70666302af7e6f28bcb1dc8d4e711e07b1b3d13"}
Sep 10 13:39:58.492912 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:58.492833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:39:58.519194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:39:58.517297    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk" podStartSLOduration=1.682413397 podStartE2EDuration="3.517275151s" totalImagesPullingTime="1.834861754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:58.516158346 +0000 UTC m=+682.971033511" watchObservedRunningTime="2026-09-10 13:39:58.517275151 +0000 UTC m=+682.972150316"
Sep 10 13:40:02.512756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:02.512664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb" event={"ID":"7f68e2ce-7a1d-47df-8e65-db2edeef86e8","Type":"ContainerStarted","Data":"55a3dbb5d93c1f37c344ec48209385238037c8cabb23a2ad89875b9b2b1edca3"}
Sep 10 13:40:02.535106 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:02.535058    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-t7glb" podStartSLOduration=1.921636451 podStartE2EDuration="6.535041578s" totalImagesPullingTime="4.613405127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:02.533637158 +0000 UTC m=+686.988512320" watchObservedRunningTime="2026-09-10 13:40:02.535041578 +0000 UTC m=+686.989916738"
Sep 10 13:40:04.459579 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:04.459550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:05.463110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:05.463078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-n7mn9"
Sep 10 13:40:07.000563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.000518    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"]
Sep 10 13:40:07.001026 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.000701    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" containerName="manager" containerID="cri-o://ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e" gracePeriod=2
Sep 10 13:40:07.011502 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.011474    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"]
Sep 10 13:40:07.011875 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.011828    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9065e732-f18e-4355-bb75-2fe20d8dd6a6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-xzxds\": pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.031305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.031279    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"]
Sep 10 13:40:07.031484 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.031463    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h" podUID="84d634a5-361c-4627-ac67-88580453fc92" containerName="manager" containerID="cri-o://ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c" gracePeriod=2
Sep 10 13:40:07.034029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.034003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.034204 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.034177    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.035411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.035353    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"]
Sep 10 13:40:07.035899 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.035866    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.037686 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.037544    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.037809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.037685    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:07.038135 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038116    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" containerName="manager"
Sep 10 13:40:07.038259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" containerName="manager"
Sep 10 13:40:07.038259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038259    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d634a5-361c-4627-ac67-88580453fc92" containerName="manager"
Sep 10 13:40:07.038430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038265    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d634a5-361c-4627-ac67-88580453fc92" containerName="manager"
Sep 10 13:40:07.038430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038353    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" containerName="manager"
Sep 10 13:40:07.038430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.038360    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d634a5-361c-4627-ac67-88580453fc92" containerName="manager"
Sep 10 13:40:07.039445 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.039397    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.043750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.043728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.046240 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.046216    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.048180 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.048158    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.058303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.058280    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"]
Sep 10 13:40:07.061689 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.061668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:07.061819 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.061773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:07.077141 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.077108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"]
Sep 10 13:40:07.090280 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.090234    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.091969 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.091939    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.179791 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.179757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdvz\" (UniqueName: \"kubernetes.io/projected/d116ee42-0148-4339-b471-a6b63d88d58b-kube-api-access-dxdvz\") pod \"limitador-operator-controller-manager-74495c69f8-lrg89\" (UID: \"d116ee42-0148-4339-b471-a6b63d88d58b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:07.180031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.179818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86xj\" (UniqueName: \"kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.180031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.179858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.266430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.266404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.268684 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.268656    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.268835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.268749    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.270455 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.270432    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.272025 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.272001    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.273654 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.273636    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.281018 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.280998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdvz\" (UniqueName: \"kubernetes.io/projected/d116ee42-0148-4339-b471-a6b63d88d58b-kube-api-access-dxdvz\") pod \"limitador-operator-controller-manager-74495c69f8-lrg89\" (UID: \"d116ee42-0148-4339-b471-a6b63d88d58b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:07.281074 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.281043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86xj\" (UniqueName: \"kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.281110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.281071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.281452 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.281435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.292260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.292233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdvz\" (UniqueName: \"kubernetes.io/projected/d116ee42-0148-4339-b471-a6b63d88d58b-kube-api-access-dxdvz\") pod \"limitador-operator-controller-manager-74495c69f8-lrg89\" (UID: \"d116ee42-0148-4339-b471-a6b63d88d58b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:07.292628 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.292613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86xj\" (UniqueName: \"kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-24qqf\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.381966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.381932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume\") pod \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") "
Sep 10 13:40:07.382135 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.382053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp2f2\" (UniqueName: \"kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2\") pod \"84d634a5-361c-4627-ac67-88580453fc92\" (UID: \"84d634a5-361c-4627-ac67-88580453fc92\") "
Sep 10 13:40:07.382135 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.382109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g549\" (UniqueName: \"kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549\") pod \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\" (UID: \"9065e732-f18e-4355-bb75-2fe20d8dd6a6\") "
Sep 10 13:40:07.382488 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.382467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9065e732-f18e-4355-bb75-2fe20d8dd6a6" (UID: "9065e732-f18e-4355-bb75-2fe20d8dd6a6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:40:07.384305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.384271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2" (OuterVolumeSpecName: "kube-api-access-fp2f2") pod "84d634a5-361c-4627-ac67-88580453fc92" (UID: "84d634a5-361c-4627-ac67-88580453fc92"). InnerVolumeSpecName "kube-api-access-fp2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:07.384305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.384289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549" (OuterVolumeSpecName: "kube-api-access-5g549") pod "9065e732-f18e-4355-bb75-2fe20d8dd6a6" (UID: "9065e732-f18e-4355-bb75-2fe20d8dd6a6"). InnerVolumeSpecName "kube-api-access-5g549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:07.426793 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.426766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:07.431947 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.431919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:07.483310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.483208    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2f2\" (UniqueName: \"kubernetes.io/projected/84d634a5-361c-4627-ac67-88580453fc92-kube-api-access-fp2f2\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:40:07.483310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.483241    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g549\" (UniqueName: \"kubernetes.io/projected/9065e732-f18e-4355-bb75-2fe20d8dd6a6-kube-api-access-5g549\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:40:07.483310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.483257    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9065e732-f18e-4355-bb75-2fe20d8dd6a6-extensions-socket-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:40:07.535362 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.535271    2577 generic.go:356] "Generic (PLEG): container finished" podID="84d634a5-361c-4627-ac67-88580453fc92" containerID="ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c" exitCode=0
Sep 10 13:40:07.535362 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.535306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.535605 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.535371    2577 scope.go:117] "RemoveContainer" containerID="ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c"
Sep 10 13:40:07.537972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.537938    2577 generic.go:356] "Generic (PLEG): container finished" podID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" containerID="ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e" exitCode=0
Sep 10 13:40:07.538397 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.538375    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.539369 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.539151    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.543462 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.541464    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.543462 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.543393    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.545128 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.545100    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.546931 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.546911    2577 scope.go:117] "RemoveContainer" containerID="ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c"
Sep 10 13:40:07.547267 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.547240    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c\": container with ID starting with ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c not found: ID does not exist" containerID="ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c"
Sep 10 13:40:07.547352 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.547277    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c"} err="failed to get container status \"ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c\": rpc error: code = NotFound desc = could not find container \"ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c\": container with ID starting with ec6c61ad01a234907a017152ea86a26b815ae35ecacb643c124958986f75ac4c not found: ID does not exist"
Sep 10 13:40:07.547352 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.547300    2577 scope.go:117] "RemoveContainer" containerID="ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e"
Sep 10 13:40:07.551844 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.551817    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.553846 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.553820    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.555140 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.555125    2577 scope.go:117] "RemoveContainer" containerID="ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e"
Sep 10 13:40:07.555411 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.555393    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e\": container with ID starting with ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e not found: ID does not exist" containerID="ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e"
Sep 10 13:40:07.555458 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.555420    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e"} err="failed to get container status \"ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e\": rpc error: code = NotFound desc = could not find container \"ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e\": container with ID starting with ad6be74e68ac650f175c92630d96a13c55b40c94f1b4d44a34970151d207798e not found: ID does not exist"
Sep 10 13:40:07.555702 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.555682    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xzxds\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xzxds"
Sep 10 13:40:07.557468 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:07.557447    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dg27h\" is forbidden: User \"system:node:ip-10-0-133-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-48.ec2.internal' and this object" podUID="84d634a5-361c-4627-ac67-88580453fc92" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dg27h"
Sep 10 13:40:07.729561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.729523    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"]
Sep 10 13:40:07.733175 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:40:07.733127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd116ee42_0148_4339_b471_a6b63d88d58b.slice/crio-b1ada2bf887ba363b06e6160b63ab00e92c47f7e7c41dd155b2650e1faf9b5ff WatchSource:0}: Error finding container b1ada2bf887ba363b06e6160b63ab00e92c47f7e7c41dd155b2650e1faf9b5ff: Status 404 returned error can't find the container with id b1ada2bf887ba363b06e6160b63ab00e92c47f7e7c41dd155b2650e1faf9b5ff
Sep 10 13:40:07.745371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:07.745350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:07.748980 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:40:07.748956    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9217a972_c172_4b1a_8bef_f2df4a8775e8.slice/crio-90dc9c54f9d6ac3eff68cd5184fd34cf938639a507804f8f228d6c963e0682a4 WatchSource:0}: Error finding container 90dc9c54f9d6ac3eff68cd5184fd34cf938639a507804f8f228d6c963e0682a4: Status 404 returned error can't find the container with id 90dc9c54f9d6ac3eff68cd5184fd34cf938639a507804f8f228d6c963e0682a4
Sep 10 13:40:08.063917 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.063833    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d634a5-361c-4627-ac67-88580453fc92" path="/var/lib/kubelet/pods/84d634a5-361c-4627-ac67-88580453fc92/volumes"
Sep 10 13:40:08.064269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.064176    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9065e732-f18e-4355-bb75-2fe20d8dd6a6" path="/var/lib/kubelet/pods/9065e732-f18e-4355-bb75-2fe20d8dd6a6/volumes"
Sep 10 13:40:08.544489 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.544451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" event={"ID":"9217a972-c172-4b1a-8bef-f2df4a8775e8","Type":"ContainerStarted","Data":"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"}
Sep 10 13:40:08.544700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.544520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" event={"ID":"9217a972-c172-4b1a-8bef-f2df4a8775e8","Type":"ContainerStarted","Data":"90dc9c54f9d6ac3eff68cd5184fd34cf938639a507804f8f228d6c963e0682a4"}
Sep 10 13:40:08.544700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.544571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:08.546127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.546093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89" event={"ID":"d116ee42-0148-4339-b471-a6b63d88d58b","Type":"ContainerStarted","Data":"75bbf4c45080136c8d8ab893a4e22b7748bcaac64700249bed41b9038f80a214"}
Sep 10 13:40:08.546231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.546140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89" event={"ID":"d116ee42-0148-4339-b471-a6b63d88d58b","Type":"ContainerStarted","Data":"b1ada2bf887ba363b06e6160b63ab00e92c47f7e7c41dd155b2650e1faf9b5ff"}
Sep 10 13:40:08.546231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.546179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:08.580869 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.580816    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" podStartSLOduration=1.5807985420000001 podStartE2EDuration="1.580798542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:08.580026478 +0000 UTC m=+693.034901642" watchObservedRunningTime="2026-09-10 13:40:08.580798542 +0000 UTC m=+693.035673760"
Sep 10 13:40:08.620239 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:08.620182    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89" podStartSLOduration=1.620166006 podStartE2EDuration="1.620166006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:08.618092108 +0000 UTC m=+693.072967271" watchObservedRunningTime="2026-09-10 13:40:08.620166006 +0000 UTC m=+693.075041169"
Sep 10 13:40:10.504855 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:10.504824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2blzk"
Sep 10 13:40:19.555366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:19.555334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lrg89"
Sep 10 13:40:19.555907 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:19.555422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:24.478853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:24.478813    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:24.479400 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:24.479106    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" podUID="9217a972-c172-4b1a-8bef-f2df4a8775e8" containerName="manager" containerID="cri-o://49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84" gracePeriod=10
Sep 10 13:40:25.223379 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.223352    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:25.334318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.334223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j86xj\" (UniqueName: \"kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj\") pod \"9217a972-c172-4b1a-8bef-f2df4a8775e8\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") "
Sep 10 13:40:25.334318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.334275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume\") pod \"9217a972-c172-4b1a-8bef-f2df4a8775e8\" (UID: \"9217a972-c172-4b1a-8bef-f2df4a8775e8\") "
Sep 10 13:40:25.334699 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.334673    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9217a972-c172-4b1a-8bef-f2df4a8775e8" (UID: "9217a972-c172-4b1a-8bef-f2df4a8775e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:40:25.336368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.336343    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj" (OuterVolumeSpecName: "kube-api-access-j86xj") pod "9217a972-c172-4b1a-8bef-f2df4a8775e8" (UID: "9217a972-c172-4b1a-8bef-f2df4a8775e8"). InnerVolumeSpecName "kube-api-access-j86xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:25.435754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.435719    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86xj\" (UniqueName: \"kubernetes.io/projected/9217a972-c172-4b1a-8bef-f2df4a8775e8-kube-api-access-j86xj\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:40:25.435754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.435752    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9217a972-c172-4b1a-8bef-f2df4a8775e8-extensions-socket-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:40:25.626911 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.626873    2577 generic.go:356] "Generic (PLEG): container finished" podID="9217a972-c172-4b1a-8bef-f2df4a8775e8" containerID="49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84" exitCode=0
Sep 10 13:40:25.627311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.626920    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"
Sep 10 13:40:25.627311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.626955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" event={"ID":"9217a972-c172-4b1a-8bef-f2df4a8775e8","Type":"ContainerDied","Data":"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"}
Sep 10 13:40:25.627311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.627009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf" event={"ID":"9217a972-c172-4b1a-8bef-f2df4a8775e8","Type":"ContainerDied","Data":"90dc9c54f9d6ac3eff68cd5184fd34cf938639a507804f8f228d6c963e0682a4"}
Sep 10 13:40:25.627311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.627034    2577 scope.go:117] "RemoveContainer" containerID="49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"
Sep 10 13:40:25.635487 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.635470    2577 scope.go:117] "RemoveContainer" containerID="49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"
Sep 10 13:40:25.635776 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:40:25.635758    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84\": container with ID starting with 49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84 not found: ID does not exist" containerID="49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"
Sep 10 13:40:25.635842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.635787    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84"} err="failed to get container status \"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84\": rpc error: code = NotFound desc = could not find container \"49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84\": container with ID starting with 49899088d26e8e0d9df2c74e9f71c01360b2d3eda7b8b95591b09af4ed920f84 not found: ID does not exist"
Sep 10 13:40:25.653582 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.653554    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:25.658268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:25.658244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-24qqf"]
Sep 10 13:40:26.063344 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:26.063261    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9217a972-c172-4b1a-8bef-f2df4a8775e8" path="/var/lib/kubelet/pods/9217a972-c172-4b1a-8bef-f2df4a8775e8/volumes"
Sep 10 13:40:41.145023 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.144984    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"]
Sep 10 13:40:41.145389 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.145308    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9217a972-c172-4b1a-8bef-f2df4a8775e8" containerName="manager"
Sep 10 13:40:41.145389 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.145319    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9217a972-c172-4b1a-8bef-f2df4a8775e8" containerName="manager"
Sep 10 13:40:41.145389 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.145372    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9217a972-c172-4b1a-8bef-f2df4a8775e8" containerName="manager"
Sep 10 13:40:41.148315 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.148294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.150620 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.150597    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kkfxl\""
Sep 10 13:40:41.150749 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.150665    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 13:40:41.163064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.163037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"]
Sep 10 13:40:41.272589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwtd\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-kube-api-access-dqwtd\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.272834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.273110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.273110 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.272928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374484 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374484 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374484 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwtd\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-kube-api-access-dqwtd\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374673 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.374926 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.375041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.374960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.375041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.375016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.376859 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.376838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.377051 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.377033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.388536 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.388498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.435708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.435629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwtd\" (UniqueName: \"kubernetes.io/projected/1dca92b2-0418-47d9-a342-0f2e4d21c55d-kube-api-access-dqwtd\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v29wf\" (UID: \"1dca92b2-0418-47d9-a342-0f2e4d21c55d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.458380 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.458348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:41.741154 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:41.741117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"]
Sep 10 13:40:41.743451 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:40:41.743425    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dca92b2_0418_47d9_a342_0f2e4d21c55d.slice/crio-9000c8b09206cc6027e17e84ff9da405eb3a7c752b33af924646a0b289ac073b WatchSource:0}: Error finding container 9000c8b09206cc6027e17e84ff9da405eb3a7c752b33af924646a0b289ac073b: Status 404 returned error can't find the container with id 9000c8b09206cc6027e17e84ff9da405eb3a7c752b33af924646a0b289ac073b
Sep 10 13:40:42.708084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:42.708043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" event={"ID":"1dca92b2-0418-47d9-a342-0f2e4d21c55d","Type":"ContainerStarted","Data":"9000c8b09206cc6027e17e84ff9da405eb3a7c752b33af924646a0b289ac073b"}
Sep 10 13:40:44.564014 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:44.563975    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:40:44.564307 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:44.564059    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:40:44.564307 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:44.564087    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:40:44.720362 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:44.720328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" event={"ID":"1dca92b2-0418-47d9-a342-0f2e4d21c55d","Type":"ContainerStarted","Data":"533bee903f3f1eb8449107b3b329ae2cecf3246b7579088cefbc2a748f6e8e47"}
Sep 10 13:40:44.748527 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:44.748458    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podStartSLOduration=0.930121209 podStartE2EDuration="3.748442963s" totalImagesPullingTime="2.818321754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:44.746292671 +0000 UTC m=+729.201167835" watchObservedRunningTime="2026-09-10 13:40:44.748442963 +0000 UTC m=+729.203318125"
Sep 10 13:40:45.459004 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.458970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:45.463668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.463641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:45.725191 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.725100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:45.726161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.726136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:40:45.827669 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.827641    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:40:45.830162 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.830141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:45.833868 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.833848    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 13:40:45.842779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.842757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:40:45.918285 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.918250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:45.918285 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.918287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xxh\" (UniqueName: \"kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:45.927666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:45.927637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:40:46.019804 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.019705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:46.019804 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.019754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xxh\" (UniqueName: \"kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:46.020559 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.020527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:46.030934 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.030907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xxh\" (UniqueName: \"kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh\") pod \"limitador-limitador-8588749977-7f2jc\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") " pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:46.140026 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.139997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:46.226161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.226131    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:40:46.229015 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.228996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.241366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.241343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:40:46.281178 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.281084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:40:46.322425 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.322389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c17b355-dbf5-43b2-8c65-1388a68da666-config-file\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.322612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.322439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnks\" (UniqueName: \"kubernetes.io/projected/4c17b355-dbf5-43b2-8c65-1388a68da666-kube-api-access-smnks\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.416472 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.416444    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:40:46.418577 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:40:46.418548    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42dba88_e342_433d_a3c3_28041ddc8e64.slice/crio-bcf453c305246f6d3a705ad67e63b9a2ee77a97e6e77dd7af487cc8c4fbf8aa5 WatchSource:0}: Error finding container bcf453c305246f6d3a705ad67e63b9a2ee77a97e6e77dd7af487cc8c4fbf8aa5: Status 404 returned error can't find the container with id bcf453c305246f6d3a705ad67e63b9a2ee77a97e6e77dd7af487cc8c4fbf8aa5
Sep 10 13:40:46.423590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.423481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c17b355-dbf5-43b2-8c65-1388a68da666-config-file\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.423590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.423568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnks\" (UniqueName: \"kubernetes.io/projected/4c17b355-dbf5-43b2-8c65-1388a68da666-kube-api-access-smnks\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.424479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.424462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4c17b355-dbf5-43b2-8c65-1388a68da666-config-file\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.433816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.433797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnks\" (UniqueName: \"kubernetes.io/projected/4c17b355-dbf5-43b2-8c65-1388a68da666-kube-api-access-smnks\") pod \"limitador-limitador-84994f7ffc-2vv9k\" (UID: \"4c17b355-dbf5-43b2-8c65-1388a68da666\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.539220 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.539144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:46.731273 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.731231    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" event={"ID":"a42dba88-e342-433d-a3c3-28041ddc8e64","Type":"ContainerStarted","Data":"bcf453c305246f6d3a705ad67e63b9a2ee77a97e6e77dd7af487cc8c4fbf8aa5"}
Sep 10 13:40:46.837008 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:46.836921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:40:46.839491 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:40:46.839460    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c17b355_dbf5_43b2_8c65_1388a68da666.slice/crio-9012f5e9e02121d61a9637477bff2800e1dbc88eca5e616d384104cea591810e WatchSource:0}: Error finding container 9012f5e9e02121d61a9637477bff2800e1dbc88eca5e616d384104cea591810e: Status 404 returned error can't find the container with id 9012f5e9e02121d61a9637477bff2800e1dbc88eca5e616d384104cea591810e
Sep 10 13:40:47.737149 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.737106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" event={"ID":"a42dba88-e342-433d-a3c3-28041ddc8e64","Type":"ContainerStarted","Data":"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"}
Sep 10 13:40:47.737655 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.737210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:47.738688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.738665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k" event={"ID":"4c17b355-dbf5-43b2-8c65-1388a68da666","Type":"ContainerStarted","Data":"f29d3dbebbead847763ff17309361a3595b5749b8f480fb3906041c7ef0d7c01"}
Sep 10 13:40:47.738800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.738707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k" event={"ID":"4c17b355-dbf5-43b2-8c65-1388a68da666","Type":"ContainerStarted","Data":"9012f5e9e02121d61a9637477bff2800e1dbc88eca5e616d384104cea591810e"}
Sep 10 13:40:47.758341 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.758293    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" podStartSLOduration=1.571580789 podStartE2EDuration="2.758278595s" totalImagesPullingTime="1.186697806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:47.757703457 +0000 UTC m=+732.212578623" watchObservedRunningTime="2026-09-10 13:40:47.758278595 +0000 UTC m=+732.213153757"
Sep 10 13:40:47.775171 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:47.775115    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k" podStartSLOduration=1.007535628 podStartE2EDuration="1.775095079s" totalImagesPullingTime="767.559451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:47.774679907 +0000 UTC m=+732.229555070" watchObservedRunningTime="2026-09-10 13:40:47.775095079 +0000 UTC m=+732.229970245"
Sep 10 13:40:48.742808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:48.742775    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:58.744082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:58.744050    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:40:59.748526 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:59.748477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"
Sep 10 13:40:59.801875 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:59.801844    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:40:59.802062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:40:59.802023    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" podUID="a42dba88-e342-433d-a3c3-28041ddc8e64" containerName="limitador" containerID="cri-o://b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826" gracePeriod=30
Sep 10 13:41:00.339362 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.339336    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:41:00.445674 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.445644    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file\") pod \"a42dba88-e342-433d-a3c3-28041ddc8e64\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") "
Sep 10 13:41:00.445861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.445722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xxh\" (UniqueName: \"kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh\") pod \"a42dba88-e342-433d-a3c3-28041ddc8e64\" (UID: \"a42dba88-e342-433d-a3c3-28041ddc8e64\") "
Sep 10 13:41:00.446043 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.446020    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file" (OuterVolumeSpecName: "config-file") pod "a42dba88-e342-433d-a3c3-28041ddc8e64" (UID: "a42dba88-e342-433d-a3c3-28041ddc8e64"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:41:00.447717 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.447693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh" (OuterVolumeSpecName: "kube-api-access-g9xxh") pod "a42dba88-e342-433d-a3c3-28041ddc8e64" (UID: "a42dba88-e342-433d-a3c3-28041ddc8e64"). InnerVolumeSpecName "kube-api-access-g9xxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:41:00.546680 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.546644    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xxh\" (UniqueName: \"kubernetes.io/projected/a42dba88-e342-433d-a3c3-28041ddc8e64-kube-api-access-g9xxh\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:41:00.546680 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.546678    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a42dba88-e342-433d-a3c3-28041ddc8e64-config-file\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:41:00.794737 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.794648    2577 generic.go:356] "Generic (PLEG): container finished" podID="a42dba88-e342-433d-a3c3-28041ddc8e64" containerID="b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826" exitCode=0
Sep 10 13:41:00.794737 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.794694    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc"
Sep 10 13:41:00.794737 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.794726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" event={"ID":"a42dba88-e342-433d-a3c3-28041ddc8e64","Type":"ContainerDied","Data":"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"}
Sep 10 13:41:00.795242 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.794769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7f2jc" event={"ID":"a42dba88-e342-433d-a3c3-28041ddc8e64","Type":"ContainerDied","Data":"bcf453c305246f6d3a705ad67e63b9a2ee77a97e6e77dd7af487cc8c4fbf8aa5"}
Sep 10 13:41:00.795242 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.794796    2577 scope.go:117] "RemoveContainer" containerID="b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"
Sep 10 13:41:00.803443 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.803250    2577 scope.go:117] "RemoveContainer" containerID="b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"
Sep 10 13:41:00.803562 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:41:00.803541    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826\": container with ID starting with b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826 not found: ID does not exist" containerID="b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"
Sep 10 13:41:00.803612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.803572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826"} err="failed to get container status \"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826\": rpc error: code = NotFound desc = could not find container \"b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826\": container with ID starting with b8fd41287259c397703a0d9a09b6b43da5406610e154594291db03bdeb499826 not found: ID does not exist"
Sep 10 13:41:00.825688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.825663    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:41:00.830381 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:00.830358    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7f2jc"]
Sep 10 13:41:02.064493 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.064457    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42dba88-e342-433d-a3c3-28041ddc8e64" path="/var/lib/kubelet/pods/a42dba88-e342-433d-a3c3-28041ddc8e64/volumes"
Sep 10 13:41:02.605575 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.605537    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"]
Sep 10 13:41:02.605871 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.605860    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42dba88-e342-433d-a3c3-28041ddc8e64" containerName="limitador"
Sep 10 13:41:02.605927 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.605873    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42dba88-e342-433d-a3c3-28041ddc8e64" containerName="limitador"
Sep 10 13:41:02.605966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.605941    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42dba88-e342-433d-a3c3-28041ddc8e64" containerName="limitador"
Sep 10 13:41:02.608576 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.608557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.618459 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.618440    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 13:41:02.618924 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.618904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jwfnp\""
Sep 10 13:41:02.618976 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.618966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 13:41:02.619051 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.618904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 13:41:02.629826 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.629801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"]
Sep 10 13:41:02.665894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.665854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpnb\" (UniqueName: \"kubernetes.io/projected/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-kube-api-access-fdpnb\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.666088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.665922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-data\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.766649 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.766606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-data\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.766827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.766696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpnb\" (UniqueName: \"kubernetes.io/projected/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-kube-api-access-fdpnb\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.767017 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.766994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-data\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.776257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.776228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpnb\" (UniqueName: \"kubernetes.io/projected/a4e831e9-f4c9-45fc-98dc-51c3fd18e10c-kube-api-access-fdpnb\") pod \"postgres-69fb4594cf-9zwrv\" (UID: \"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:02.918564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:02.918502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:03.205207 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:03.205081    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"]
Sep 10 13:41:03.207750 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:41:03.207723    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e831e9_f4c9_45fc_98dc_51c3fd18e10c.slice/crio-0d94ccf163e7ba53e01faf9f3c80eca7cbdba867832bbd46b689f33b79b53496 WatchSource:0}: Error finding container 0d94ccf163e7ba53e01faf9f3c80eca7cbdba867832bbd46b689f33b79b53496: Status 404 returned error can't find the container with id 0d94ccf163e7ba53e01faf9f3c80eca7cbdba867832bbd46b689f33b79b53496
Sep 10 13:41:03.809798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:03.809751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv" event={"ID":"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c","Type":"ContainerStarted","Data":"0d94ccf163e7ba53e01faf9f3c80eca7cbdba867832bbd46b689f33b79b53496"}
Sep 10 13:41:05.821612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:05.821498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv" event={"ID":"a4e831e9-f4c9-45fc-98dc-51c3fd18e10c","Type":"ContainerStarted","Data":"f1a50b4e561bc955e6e6b7f826772ecc9327b1987197044587f7949a02f43f4f"}
Sep 10 13:41:05.821997 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:05.821653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:05.844911 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:05.844856    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv" podStartSLOduration=1.62271202 podStartE2EDuration="3.844839972s" totalImagesPullingTime="2.222127952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:05.843581294 +0000 UTC m=+750.298456457" watchObservedRunningTime="2026-09-10 13:41:05.844839972 +0000 UTC m=+750.299715135"
Sep 10 13:41:11.853870 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:11.853839    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-9zwrv"
Sep 10 13:41:12.629366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.629334    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-frr7g"]
Sep 10 13:41:12.635923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.635903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.639026 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.639003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 13:41:12.639139 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.639003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6r98w\""
Sep 10 13:41:12.645220 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.645196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-frr7g"]
Sep 10 13:41:12.656048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.656013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdj4l\" (UniqueName: \"kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.656180 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.656078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.704897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.704863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-frr7g"]
Sep 10 13:41:12.705136 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:41:12.705118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rdj4l tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-frr7g" podUID="648ad9a6-1cb2-40b8-9393-3332b331b5eb"
Sep 10 13:41:12.756678 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.756640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdj4l\" (UniqueName: \"kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.756873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.756690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.759023 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.758999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.791820 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.791790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdj4l\" (UniqueName: \"kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l\") pod \"authorino-6d4dfc649f-frr7g\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") " pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.851568 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.851539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.859406 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.859381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:12.957811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.957721    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdj4l\" (UniqueName: \"kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l\") pod \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") "
Sep 10 13:41:12.957811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.957790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert\") pod \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\" (UID: \"648ad9a6-1cb2-40b8-9393-3332b331b5eb\") "
Sep 10 13:41:12.959760 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.959737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l" (OuterVolumeSpecName: "kube-api-access-rdj4l") pod "648ad9a6-1cb2-40b8-9393-3332b331b5eb" (UID: "648ad9a6-1cb2-40b8-9393-3332b331b5eb"). InnerVolumeSpecName "kube-api-access-rdj4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:41:12.959842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:12.959826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "648ad9a6-1cb2-40b8-9393-3332b331b5eb" (UID: "648ad9a6-1cb2-40b8-9393-3332b331b5eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:41:13.058859 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:13.058828    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdj4l\" (UniqueName: \"kubernetes.io/projected/648ad9a6-1cb2-40b8-9393-3332b331b5eb-kube-api-access-rdj4l\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:41:13.058859 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:13.058854    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/648ad9a6-1cb2-40b8-9393-3332b331b5eb-tls-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:41:13.855497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:13.855470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-frr7g"
Sep 10 13:41:13.890772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:13.890737    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-frr7g"]
Sep 10 13:41:13.895467 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:13.895439    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-frr7g"]
Sep 10 13:41:14.063060 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:14.063029    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648ad9a6-1cb2-40b8-9393-3332b331b5eb" path="/var/lib/kubelet/pods/648ad9a6-1cb2-40b8-9393-3332b331b5eb/volumes"
Sep 10 13:41:56.463298 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.463262    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 13:41:56.468194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.468172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.470474 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.470453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q8qnd\""
Sep 10 13:41:56.470718 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.470703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 13:41:56.470770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.470729    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 13:41:56.479998 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.479976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 13:41:56.540147 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.540107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.540342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.540213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpj7d\" (UniqueName: \"kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.540342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.540246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.540342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.540289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.641663 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.641620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpj7d\" (UniqueName: \"kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.641663 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.641670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.641915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.641716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.641915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.641766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.642432 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.642407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.644185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.644165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.644310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.644289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.651481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.651458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpj7d\" (UniqueName: \"kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d\") pod \"maas-api-cb9d756dc-9lfqk\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") " pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:56.777963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:56.777877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:41:57.057236 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:57.057208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 13:41:57.059878 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:41:57.059848    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb2670e_2045_4af3_b1db_00c4fa7faf8b.slice/crio-fc712267ee6908ab17b0f663f7326d647f5e40cb658bbbf1051e748f24fe01d6 WatchSource:0}: Error finding container fc712267ee6908ab17b0f663f7326d647f5e40cb658bbbf1051e748f24fe01d6: Status 404 returned error can't find the container with id fc712267ee6908ab17b0f663f7326d647f5e40cb658bbbf1051e748f24fe01d6
Sep 10 13:41:57.264612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:57.264572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:41:58.063918 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:41:58.063876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" event={"ID":"8fb2670e-2045-4af3-b1db-00c4fa7faf8b","Type":"ContainerStarted","Data":"fc712267ee6908ab17b0f663f7326d647f5e40cb658bbbf1051e748f24fe01d6"}
Sep 10 13:42:01.080462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:01.080354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" event={"ID":"8fb2670e-2045-4af3-b1db-00c4fa7faf8b","Type":"ContainerStarted","Data":"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"}
Sep 10 13:42:01.080462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:01.080455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:42:01.100624 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:01.100569    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" podStartSLOduration=1.4995114 podStartE2EDuration="5.100550582s" totalImagesPullingTime="3.601039182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:42:01.099527646 +0000 UTC m=+805.554402821" watchObservedRunningTime="2026-09-10 13:42:01.100550582 +0000 UTC m=+805.555425764"
Sep 10 13:42:07.090084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.090049    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 13:42:07.109567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.109503    2577 generic.go:356] "Generic (PLEG): container finished" podID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerID="533bee903f3f1eb8449107b3b329ae2cecf3246b7579088cefbc2a748f6e8e47" exitCode=0
Sep 10 13:42:07.109842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.109580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" event={"ID":"1dca92b2-0418-47d9-a342-0f2e4d21c55d","Type":"ContainerDied","Data":"533bee903f3f1eb8449107b3b329ae2cecf3246b7579088cefbc2a748f6e8e47"}
Sep 10 13:42:07.110483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.110116    2577 scope.go:117] "RemoveContainer" containerID="533bee903f3f1eb8449107b3b329ae2cecf3246b7579088cefbc2a748f6e8e47"
Sep 10 13:42:07.111291 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.111263    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:42:07.111347 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.111327    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:42:07.111389 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:07.111360    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:42:08.115754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:08.115719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" event={"ID":"1dca92b2-0418-47d9-a342-0f2e4d21c55d","Type":"ContainerStarted","Data":"d59ff3d1b24a17052b74eb8d6c0b81acd10dcef110d677f2e58037f205af4e5b"}
Sep 10 13:42:08.459368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:08.459258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:42:08.459762 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:08.459728    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v29wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 10 13:42:08.459881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:08.459789    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podUID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 10 13:42:09.460772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:09.460243    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v29wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 10 13:42:09.460772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:09.460312    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podUID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 10 13:42:10.459551 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:10.459490    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v29wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 10 13:42:10.459771 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:10.459579    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podUID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 10 13:42:11.459175 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:11.459127    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v29wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 10 13:42:11.459708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:11.459200    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podUID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 10 13:42:12.458837 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:12.458797    2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v29wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 10 13:42:12.459155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:12.459123    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf" podUID="1dca92b2-0418-47d9-a342-0f2e4d21c55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 10 13:42:13.335723 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:13.335681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:42:13.462591 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:13.462556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:42:13.462844 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:13.462819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:42:13.463428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:13.463413    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v29wf"
Sep 10 13:42:29.644491 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:29.644451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:42:57.892647 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:42:57.892606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:43:09.683699 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.683663    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"]
Sep 10 13:43:09.687145 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.687121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.689701 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.689671    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vzhxx\""
Sep 10 13:43:09.689834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.689756    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 13:43:09.689983 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.689966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 13:43:09.690633 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.690619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 13:43:09.698139 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.698116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"]
Sep 10 13:43:09.785653 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc95104-471f-41a2-afea-ab36f7a614d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.785827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.785827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.785827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wr7\" (UniqueName: \"kubernetes.io/projected/cfc95104-471f-41a2-afea-ab36f7a614d3-kube-api-access-r6wr7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.785827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.785827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.785807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886730 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886730 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wr7\" (UniqueName: \"kubernetes.io/projected/cfc95104-471f-41a2-afea-ab36f7a614d3-kube-api-access-r6wr7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886890 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc95104-471f-41a2-afea-ab36f7a614d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886940 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.886996 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.886960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.887089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.887073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.889084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.889063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc95104-471f-41a2-afea-ab36f7a614d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.889322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.889306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc95104-471f-41a2-afea-ab36f7a614d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.896194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.896170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wr7\" (UniqueName: \"kubernetes.io/projected/cfc95104-471f-41a2-afea-ab36f7a614d3-kube-api-access-r6wr7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9jlr8\" (UID: \"cfc95104-471f-41a2-afea-ab36f7a614d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:09.997206 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:09.997119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:10.270666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:10.270595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"]
Sep 10 13:43:10.272899 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:43:10.272871    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc95104_471f_41a2_afea_ab36f7a614d3.slice/crio-5297e0c1baa7e19aa10be12cb136d135106c5d6040d9ce0b3c976d10a21d772f WatchSource:0}: Error finding container 5297e0c1baa7e19aa10be12cb136d135106c5d6040d9ce0b3c976d10a21d772f: Status 404 returned error can't find the container with id 5297e0c1baa7e19aa10be12cb136d135106c5d6040d9ce0b3c976d10a21d772f
Sep 10 13:43:10.274276 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:10.274258    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:43:10.396953 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:10.396915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8" event={"ID":"cfc95104-471f-41a2-afea-ab36f7a614d3","Type":"ContainerStarted","Data":"5297e0c1baa7e19aa10be12cb136d135106c5d6040d9ce0b3c976d10a21d772f"}
Sep 10 13:43:10.943237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:10.943203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:43:11.404270 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:11.404233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8" event={"ID":"cfc95104-471f-41a2-afea-ab36f7a614d3","Type":"ContainerStarted","Data":"5382e6488766b1584deffb282699c8e9a69e3813434ebba1fb4d37ceb84fa93d"}
Sep 10 13:43:11.446560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:11.446486    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8" podStartSLOduration=1.441805316 podStartE2EDuration="2.446464653s" totalImagesPullingTime="1.004659337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:11.44603416 +0000 UTC m=+875.900909324" watchObservedRunningTime="2026-09-10 13:43:11.446464653 +0000 UTC m=+875.901339816"
Sep 10 13:43:16.713073 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:16.713037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:43:18.995959 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:18.995926    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"]
Sep 10 13:43:19.000603 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.000574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.003132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.003110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 13:43:19.009419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.009394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"]
Sep 10 13:43:19.070172 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.070353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.070353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcts4\" (UniqueName: \"kubernetes.io/projected/0a08ade6-203f-42ae-9224-ef0f5495e5ab-kube-api-access-gcts4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.070353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.070353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.070353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.070347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.170792 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.170759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.170792 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.170797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.170825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcts4\" (UniqueName: \"kubernetes.io/projected/0a08ade6-203f-42ae-9224-ef0f5495e5ab-kube-api-access-gcts4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.170957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.171006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.171031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.171358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171493 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.171420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.171569 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.171494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.173634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.173610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08ade6-203f-42ae-9224-ef0f5495e5ab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.174011 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.173987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08ade6-203f-42ae-9224-ef0f5495e5ab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.183785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.181674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcts4\" (UniqueName: \"kubernetes.io/projected/0a08ade6-203f-42ae-9224-ef0f5495e5ab-kube-api-access-gcts4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cfvpx\" (UID: \"0a08ade6-203f-42ae-9224-ef0f5495e5ab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.312184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.312086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:19.599129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.599037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"]
Sep 10 13:43:19.602195 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:43:19.602166    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a08ade6_203f_42ae_9224_ef0f5495e5ab.slice/crio-c3d24f79fe56c1a8cc3158097884ed7d855b3dd692f5a1e85033e4e5241dd19f WatchSource:0}: Error finding container c3d24f79fe56c1a8cc3158097884ed7d855b3dd692f5a1e85033e4e5241dd19f: Status 404 returned error can't find the container with id c3d24f79fe56c1a8cc3158097884ed7d855b3dd692f5a1e85033e4e5241dd19f
Sep 10 13:43:19.997798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.997759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:19.998248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:19.997827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:20.010591 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:20.010563    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:20.449835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:20.449788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx" event={"ID":"0a08ade6-203f-42ae-9224-ef0f5495e5ab","Type":"ContainerStarted","Data":"c37a55bb58bcbfc25933009dc2802cc4ddb72add84cbce9abc00c4f493f547f6"}
Sep 10 13:43:20.450048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:20.449855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx" event={"ID":"0a08ade6-203f-42ae-9224-ef0f5495e5ab","Type":"ContainerStarted","Data":"c3d24f79fe56c1a8cc3158097884ed7d855b3dd692f5a1e85033e4e5241dd19f"}
Sep 10 13:43:20.461619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:20.461590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9jlr8"
Sep 10 13:43:20.468947 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:20.468897    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx" podStartSLOduration=2.215138915 podStartE2EDuration="2.46888091s" totalImagesPullingTime="253.741995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:20.467780971 +0000 UTC m=+884.922656135" watchObservedRunningTime="2026-09-10 13:43:20.46888091 +0000 UTC m=+884.923756106"
Sep 10 13:43:21.038292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:21.038255    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:43:24.701630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.701594    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"]
Sep 10 13:43:24.705991 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.705961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.708466 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.708443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 13:43:24.716339 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.716313    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"]
Sep 10 13:43:24.827735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.827936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7gv\" (UniqueName: \"kubernetes.io/projected/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-kube-api-access-jn7gv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.827936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.827936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.828084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.828084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.827979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928710 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7gv\" (UniqueName: \"kubernetes.io/projected/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-kube-api-access-jn7gv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.928906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.928897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.929631 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.929599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.929832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.929662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.929936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.929725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.932212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.932177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.932520 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.932484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:24.947277 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:24.947244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7gv\" (UniqueName: \"kubernetes.io/projected/0e8e1d2d-9318-4e9b-9638-800859c6eaf6-kube-api-access-jn7gv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6pplx\" (UID: \"0e8e1d2d-9318-4e9b-9638-800859c6eaf6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:25.037769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:25.037675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:25.312889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:25.312861    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"]
Sep 10 13:43:25.314773 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:43:25.314746    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e8e1d2d_9318_4e9b_9638_800859c6eaf6.slice/crio-919c0cb99df6aa96f2338befa5329b01298a4fda00b60ec85b3f08fa99b0abd6 WatchSource:0}: Error finding container 919c0cb99df6aa96f2338befa5329b01298a4fda00b60ec85b3f08fa99b0abd6: Status 404 returned error can't find the container with id 919c0cb99df6aa96f2338befa5329b01298a4fda00b60ec85b3f08fa99b0abd6
Sep 10 13:43:25.474946 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:25.474908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx" event={"ID":"0e8e1d2d-9318-4e9b-9638-800859c6eaf6","Type":"ContainerStarted","Data":"919c0cb99df6aa96f2338befa5329b01298a4fda00b60ec85b3f08fa99b0abd6"}
Sep 10 13:43:26.481607 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:26.481566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx" event={"ID":"0e8e1d2d-9318-4e9b-9638-800859c6eaf6","Type":"ContainerStarted","Data":"427d539a9c1a8e8c442b5659475f308920b773f858558ecfd992b7b6526a6da3"}
Sep 10 13:43:26.504560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:26.504483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx" podStartSLOduration=2.293019894 podStartE2EDuration="2.504468107s" totalImagesPullingTime="211.448213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:43:26.50398252 +0000 UTC m=+890.958857687" watchObservedRunningTime="2026-09-10 13:43:26.504468107 +0000 UTC m=+890.959343281"
Sep 10 13:43:26.734706 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:26.734617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:43:29.312826 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:29.312739    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:29.312826 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:29.312806    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:29.325406 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:29.325378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:29.507210 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:29.507178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cfvpx"
Sep 10 13:43:35.038580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:35.038537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:35.039028 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:35.038596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:35.051606 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:35.051579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:35.533785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:35.533756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6pplx"
Sep 10 13:43:36.039136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:36.039105    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:43:36.039892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:43:36.039871    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:45:36.863297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:36.863263    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:45:38.147213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.147178    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:45:38.151146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.151119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.153244 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.153221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-86fdcf-serving-cert\""
Sep 10 13:45:38.164644 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.164612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:45:38.239020 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.238983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng65d\" (UniqueName: \"kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.239197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.239087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.239197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.239123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.239273 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.239188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.340440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.340388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng65d\" (UniqueName: \"kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.340676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.340520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.340676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.340570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.340676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.340663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.340993 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:45:38.340967    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-serving-cert: secret "maas-api-e2e-inf-a-86fdcf-serving-cert" not found
Sep 10 13:45:38.341072 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:45:38.341057    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls podName:9759f70e-dacb-45d7-be59-ca0a33dc4df3 nodeName:}" failed. No retries permitted until 2026-09-10 13:45:38.841031828 +0000 UTC m=+1023.295906986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls") pod "maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" (UID: "9759f70e-dacb-45d7-be59-ca0a33dc4df3") : secret "maas-api-e2e-inf-a-86fdcf-serving-cert" not found
Sep 10 13:45:38.341529 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.341476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.343837 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.343802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.359146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.359109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng65d\" (UniqueName: \"kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.846923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.846885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:38.849293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:38.849270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") pod \"maas-api-e2e-inf-a-86fdcf-8564c45858-kl622\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:39.062611 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:39.062569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:39.336428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:39.336398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:45:39.339385 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:45:39.339352    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9759f70e_dacb_45d7_be59_ca0a33dc4df3.slice/crio-ed7ff474d1b2696e6b93f1f6b7f4c601355eadb190dfcfdb18df36b4061f0432 WatchSource:0}: Error finding container ed7ff474d1b2696e6b93f1f6b7f4c601355eadb190dfcfdb18df36b4061f0432: Status 404 returned error can't find the container with id ed7ff474d1b2696e6b93f1f6b7f4c601355eadb190dfcfdb18df36b4061f0432
Sep 10 13:45:40.099549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:40.099496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" event={"ID":"9759f70e-dacb-45d7-be59-ca0a33dc4df3","Type":"ContainerStarted","Data":"a56d46353d459f0aa5543c793c35465c4bb190d20f68f54721dc5dca456ff3ec"}
Sep 10 13:45:40.099758 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:40.099574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" event={"ID":"9759f70e-dacb-45d7-be59-ca0a33dc4df3","Type":"ContainerStarted","Data":"ed7ff474d1b2696e6b93f1f6b7f4c601355eadb190dfcfdb18df36b4061f0432"}
Sep 10 13:45:40.099758 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:40.099600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:40.130375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:40.130312    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" podStartSLOduration=1.561597746 podStartE2EDuration="2.130291457s" totalImagesPullingTime="568.693711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:40.129255353 +0000 UTC m=+1024.584130517" watchObservedRunningTime="2026-09-10 13:45:40.130291457 +0000 UTC m=+1024.585166614"
Sep 10 13:45:43.875766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.875726    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:45:43.879205 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.879182    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.884013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.883990    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-854ac94d-openshift-default-dockercfg-cb84x\""
Sep 10 13:45:43.897025 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.896998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:45:43.996152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnsj\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:43.996609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:43.996434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.097697 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.097697 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.097950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.097950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.097950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098120 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.097949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnsj\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098120 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098120 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098120 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098456 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.098769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.098731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.100159 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.100133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.100372 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.100354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.112003 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.111980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.112877 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.112843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnsj\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj\") pod \"e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") " pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.189227 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.189133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:44.469789 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.469760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:45:44.472421 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:45:44.472395    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ad718a_2cea_4538_9cb2_3a5f9fb558cc.slice/crio-0ab58251ba633f4f4cb8831a712d9af48ac4ab172b134a92b8b0be7b50f077d4 WatchSource:0}: Error finding container 0ab58251ba633f4f4cb8831a712d9af48ac4ab172b134a92b8b0be7b50f077d4: Status 404 returned error can't find the container with id 0ab58251ba633f4f4cb8831a712d9af48ac4ab172b134a92b8b0be7b50f077d4
Sep 10 13:45:44.474756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.474724    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:45:44.474806 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.474792    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:45:44.474846 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:44.474820    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:45:45.124437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:45.124400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerStarted","Data":"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"}
Sep 10 13:45:45.124889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:45.124458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerStarted","Data":"0ab58251ba633f4f4cb8831a712d9af48ac4ab172b134a92b8b0be7b50f077d4"}
Sep 10 13:45:45.158081 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:45.158005    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podStartSLOduration=2.157980145 podStartE2EDuration="2.157980145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:45.154421558 +0000 UTC m=+1029.609296736" watchObservedRunningTime="2026-09-10 13:45:45.157980145 +0000 UTC m=+1029.612855308"
Sep 10 13:45:45.189940 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:45.189890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:45.195880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:45.195843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:46.101796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.101766    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:45:46.105435 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.105403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.107598 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.107575    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b1f490-serving-cert\""
Sep 10 13:45:46.111003 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.110981    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:45:46.117279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.117252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:45:46.129549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.129525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:46.130600 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.130583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:45:46.218478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.218441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.219084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.219048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.219203 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.219175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.219535 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.219487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf86b\" (UniqueName: \"kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.321822 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.321775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf86b\" (UniqueName: \"kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.322036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.321859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.322036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.321896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.322036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.321943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.322036 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:45:46.322002    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-serving-cert: secret "maas-api-e2e-autores-b1f490-serving-cert" not found
Sep 10 13:45:46.322244 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:45:46.322086    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls podName:62e94d02-4393-49a2-b743-8dd9657d69bd nodeName:}" failed. No retries permitted until 2026-09-10 13:45:46.82206416 +0000 UTC m=+1031.276939300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls") pod "maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" (UID: "62e94d02-4393-49a2-b743-8dd9657d69bd") : secret "maas-api-e2e-autores-b1f490-serving-cert" not found
Sep 10 13:45:46.322640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.322618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.324484 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.324460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.331715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.331691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf86b\" (UniqueName: \"kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.826807 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.826773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:46.829587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:46.829558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") pod \"maas-api-e2e-autores-b1f490-6b74957c77-5vqd8\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:47.018578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:47.018544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:47.318789 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:47.318754    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:45:47.322310 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:45:47.322284    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e94d02_4393_49a2_b743_8dd9657d69bd.slice/crio-d04f25b484fbcd533c53e822bbda38a8fd2f96d39d1ed115c142124952a42756 WatchSource:0}: Error finding container d04f25b484fbcd533c53e822bbda38a8fd2f96d39d1ed115c142124952a42756: Status 404 returned error can't find the container with id d04f25b484fbcd533c53e822bbda38a8fd2f96d39d1ed115c142124952a42756
Sep 10 13:45:48.141705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.141674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" event={"ID":"62e94d02-4393-49a2-b743-8dd9657d69bd","Type":"ContainerStarted","Data":"d04f25b484fbcd533c53e822bbda38a8fd2f96d39d1ed115c142124952a42756"}
Sep 10 13:45:48.499972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.499939    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:45:48.503468 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.503447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.514996 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.514969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-854ac94d-serving-cert\""
Sep 10 13:45:48.596987 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.596951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:45:48.644383 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.644351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.644612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.644414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njgzg\" (UniqueName: \"kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.644612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.644498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.644612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.644555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.745439 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.745398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.745666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.745462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njgzg\" (UniqueName: \"kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.745666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.745495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.745666 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.745542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.746224 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.746199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.748065 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.748042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.748156 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.748080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.760617 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.760556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njgzg\" (UniqueName: \"kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg\") pod \"maas-api-e2e-ait-854ac94d-8489fcf574-swxmm\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:48.813664 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:48.813625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:49.111748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:49.111720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:45:49.112920 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:45:49.112892    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9a9419_e9bd_4f58_a969_f9863fe325f3.slice/crio-459557e29178c1a16b0a04624563c12e27e0b35cd1cafca78ce0bda18668f16a WatchSource:0}: Error finding container 459557e29178c1a16b0a04624563c12e27e0b35cd1cafca78ce0bda18668f16a: Status 404 returned error can't find the container with id 459557e29178c1a16b0a04624563c12e27e0b35cd1cafca78ce0bda18668f16a
Sep 10 13:45:49.148111 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:49.148066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" event={"ID":"bc9a9419-e9bd-4f58-a969-f9863fe325f3","Type":"ContainerStarted","Data":"459557e29178c1a16b0a04624563c12e27e0b35cd1cafca78ce0bda18668f16a"}
Sep 10 13:45:49.149677 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:49.149650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" event={"ID":"62e94d02-4393-49a2-b743-8dd9657d69bd","Type":"ContainerStarted","Data":"01c13ce4c7006f9423fd74d56c9052728b984e417df0eaf71caef0f039818cde"}
Sep 10 13:45:49.149816 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:49.149761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:49.179244 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:49.179180    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" podStartSLOduration=2.413207883 podStartE2EDuration="3.179164603s" totalImagesPullingTime="765.95672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:49.177314303 +0000 UTC m=+1033.632189466" watchObservedRunningTime="2026-09-10 13:45:49.179164603 +0000 UTC m=+1033.634039766"
Sep 10 13:45:50.158811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:50.158762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" event={"ID":"bc9a9419-e9bd-4f58-a969-f9863fe325f3","Type":"ContainerStarted","Data":"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"}
Sep 10 13:45:50.159215 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:50.158844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:45:50.191444 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:50.191392    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" podStartSLOduration=1.829085496 podStartE2EDuration="2.191374096s" totalImagesPullingTime="362.2886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:50.191313817 +0000 UTC m=+1034.646188980" watchObservedRunningTime="2026-09-10 13:45:50.191374096 +0000 UTC m=+1034.646249237"
Sep 10 13:45:51.648721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:51.648688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:45:55.164440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:55.164410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:45:56.168705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:45:56.168676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:46:01.692933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:01.692669    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:47724->10.134.0.38:15021: read: connection reset by peer" start-of-body=
Sep 10 13:46:01.692933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:01.692745    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:47724->10.134.0.38:15021: read: connection reset by peer"
Sep 10 13:46:02.217039 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.217003    2577 generic.go:356] "Generic (PLEG): container finished" podID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerID="c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4" exitCode=0
Sep 10 13:46:02.217216 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.217062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerDied","Data":"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"}
Sep 10 13:46:02.217564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.217545    2577 scope.go:117] "RemoveContainer" containerID="c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"
Sep 10 13:46:02.218664 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.218632    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:46:02.218751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.218718    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:46:02.218787 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:02.218756    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:46:03.224409 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:03.224375    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerStarted","Data":"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"}
Sep 10 13:46:04.190069 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:04.190022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:04.190342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:04.190317    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 13:46:04.190400 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:04.190370    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 13:46:05.190088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:05.190044    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 13:46:05.190480 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:05.190109    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 13:46:06.190094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:06.190055    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 13:46:06.190475 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:06.190119    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 13:46:07.190395 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:07.190357    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 13:46:07.190838 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:07.190414    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 13:46:08.189815 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.189771    2577 patch_prober.go:28] interesting pod/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 10 13:46:08.190119 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.189839    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 10 13:46:08.280931 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.280888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:46:08.771883 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.771840    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:46:08.775798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.775771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.779775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.779751    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-02fc2e-serving-cert\""
Sep 10 13:46:08.813152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.813124    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:46:08.838353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.838317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.838556 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.838387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.838556 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.838421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dpr\" (UniqueName: \"kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.838556 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.838503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.939423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.939389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.939646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.939432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dpr\" (UniqueName: \"kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.939646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.939479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.939646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.939571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.939819 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:08.939705    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-serving-cert: secret "maas-api-e2e-inf-b-02fc2e-serving-cert" not found
Sep 10 13:46:08.939819 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:08.939775    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls podName:2e828ce8-fa01-459c-add8-19566d3f8217 nodeName:}" failed. No retries permitted until 2026-09-10 13:46:09.439754373 +0000 UTC m=+1053.894629532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls") pod "maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" (UID: "2e828ce8-fa01-459c-add8-19566d3f8217") : secret "maas-api-e2e-inf-b-02fc2e-serving-cert" not found
Sep 10 13:46:08.940250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.940229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.941882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.941854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:08.956837 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:08.956803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dpr\" (UniqueName: \"kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:09.193624 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.193591    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:09.193897 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.193876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:09.194490 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.194475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:09.444143 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.444063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:09.446418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.446389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") pod \"maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:09.685800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.685764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:09.972132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:09.972107    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:46:09.974092 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:46:09.974061    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e828ce8_fa01_459c_add8_19566d3f8217.slice/crio-a95721d20fdf867d334436c9c7293685856cf6201cfd49fc77149c56a0856e53 WatchSource:0}: Error finding container a95721d20fdf867d334436c9c7293685856cf6201cfd49fc77149c56a0856e53: Status 404 returned error can't find the container with id a95721d20fdf867d334436c9c7293685856cf6201cfd49fc77149c56a0856e53
Sep 10 13:46:10.267925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:10.267840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" event={"ID":"2e828ce8-fa01-459c-add8-19566d3f8217","Type":"ContainerStarted","Data":"a95721d20fdf867d334436c9c7293685856cf6201cfd49fc77149c56a0856e53"}
Sep 10 13:46:11.273867 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:11.273831    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" event={"ID":"2e828ce8-fa01-459c-add8-19566d3f8217","Type":"ContainerStarted","Data":"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"}
Sep 10 13:46:11.274353 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:11.273932    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:11.312820 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:11.312767    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" podStartSLOduration=2.813119051 podStartE2EDuration="3.312749413s" totalImagesPullingTime="499.630362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:11.306983959 +0000 UTC m=+1055.761859121" watchObservedRunningTime="2026-09-10 13:46:11.312749413 +0000 UTC m=+1055.767624584"
Sep 10 13:46:17.284352 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:17.284321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:46:22.269123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:22.269088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:46:28.817982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:28.817896    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:46:28.818521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:28.818107    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" podUID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" containerName="maas-api" containerID="cri-o://604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763" gracePeriod=30
Sep 10 13:46:29.065797 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.065769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:46:29.250204 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njgzg\" (UniqueName: \"kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg\") pod \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") "
Sep 10 13:46:29.250371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls\") pod \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") "
Sep 10 13:46:29.250371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs\") pod \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") "
Sep 10 13:46:29.250371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca\") pod \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\" (UID: \"bc9a9419-e9bd-4f58-a969-f9863fe325f3\") "
Sep 10 13:46:29.251451 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc9a9419-e9bd-4f58-a969-f9863fe325f3" (UID: "bc9a9419-e9bd-4f58-a969-f9863fe325f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:29.251451 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.250914    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9a9419-e9bd-4f58-a969-f9863fe325f3-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:29.253002 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.252966    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg" (OuterVolumeSpecName: "kube-api-access-njgzg") pod "bc9a9419-e9bd-4f58-a969-f9863fe325f3" (UID: "bc9a9419-e9bd-4f58-a969-f9863fe325f3"). InnerVolumeSpecName "kube-api-access-njgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:29.253092 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.253020    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc9a9419-e9bd-4f58-a969-f9863fe325f3" (UID: "bc9a9419-e9bd-4f58-a969-f9863fe325f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:29.253092 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.253023    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc9a9419-e9bd-4f58-a969-f9863fe325f3" (UID: "bc9a9419-e9bd-4f58-a969-f9863fe325f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:29.352107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.352061    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njgzg\" (UniqueName: \"kubernetes.io/projected/bc9a9419-e9bd-4f58-a969-f9863fe325f3-kube-api-access-njgzg\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:29.352252 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.352111    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:29.352252 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.352127    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc9a9419-e9bd-4f58-a969-f9863fe325f3-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:29.365992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.365957    2577 generic.go:356] "Generic (PLEG): container finished" podID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" containerID="604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763" exitCode=0
Sep 10 13:46:29.366179 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.366030    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"
Sep 10 13:46:29.366179 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.366048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" event={"ID":"bc9a9419-e9bd-4f58-a969-f9863fe325f3","Type":"ContainerDied","Data":"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"}
Sep 10 13:46:29.366179 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.366153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm" event={"ID":"bc9a9419-e9bd-4f58-a969-f9863fe325f3","Type":"ContainerDied","Data":"459557e29178c1a16b0a04624563c12e27e0b35cd1cafca78ce0bda18668f16a"}
Sep 10 13:46:29.366343 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.366188    2577 scope.go:117] "RemoveContainer" containerID="604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"
Sep 10 13:46:29.378354 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.378332    2577 scope.go:117] "RemoveContainer" containerID="604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"
Sep 10 13:46:29.378708 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:29.378688    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763\": container with ID starting with 604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763 not found: ID does not exist" containerID="604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"
Sep 10 13:46:29.378781 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.378722    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763"} err="failed to get container status \"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763\": rpc error: code = NotFound desc = could not find container \"604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763\": container with ID starting with 604e477b07b6029b14dff51ecdbe1b03af46fd1885f8d286cc2fc8085dc71763 not found: ID does not exist"
Sep 10 13:46:29.389829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.389802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:46:29.390011 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.389991    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy" containerID="cri-o://bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d" gracePeriod=30
Sep 10 13:46:29.395817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.395794    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:46:29.399713 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:29.399690    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-854ac94d-8489fcf574-swxmm"]
Sep 10 13:46:30.063661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:30.063624    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" path="/var/lib/kubelet/pods/bc9a9419-e9bd-4f58-a969-f9863fe325f3/volumes"
Sep 10 13:46:33.844617 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.844575    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:46:33.845165 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.845144    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" containerName="maas-api"
Sep 10 13:46:33.845260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.845168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" containerName="maas-api"
Sep 10 13:46:33.845321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.845301    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9a9419-e9bd-4f58-a969-f9863fe325f3" containerName="maas-api"
Sep 10 13:46:33.849644 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.849621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.852282 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.852256    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0e325272-serving-cert\""
Sep 10 13:46:33.860346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.860318    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:46:33.892450 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.892417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.892669 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.892479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xvj\" (UniqueName: \"kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.892669 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.892615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.892790 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.892681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.993636 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.993599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.993849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.993672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.993849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.993747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xvj\" (UniqueName: \"kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.993849 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:33.993772    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-serving-cert: secret "maas-api-e2e-migrate-0e325272-serving-cert" not found
Sep 10 13:46:33.993849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.993809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.993849 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:33.993849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls podName:4b0cf199-014f-4541-979d-a848e99e416d nodeName:}" failed. No retries permitted until 2026-09-10 13:46:34.493832638 +0000 UTC m=+1078.948707779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls") pod "maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" (UID: "4b0cf199-014f-4541-979d-a848e99e416d") : secret "maas-api-e2e-migrate-0e325272-serving-cert" not found
Sep 10 13:46:33.994454 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.994431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:33.996044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:33.996026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:34.003253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.003228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xvj\" (UniqueName: \"kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:34.498586 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.498500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:34.501014 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.500980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") pod \"maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:34.669723 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.669694    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:34.700268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700365    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhnsj\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:34.700908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs\") pod \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\" (UID: \"58ad718a-2cea-4538-9cb2-3a5f9fb558cc\") "
Sep 10 13:46:34.700908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:34.701125 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700941    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.701125 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.700962    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.701125 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.701060    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:34.701524 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.701482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:34.701763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.701742    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:34.703290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.703268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:34.703418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.703400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:46:34.703492 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.703464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:46:34.703587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.703471    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj" (OuterVolumeSpecName: "kube-api-access-bhnsj") pod "58ad718a-2cea-4538-9cb2-3a5f9fb558cc" (UID: "58ad718a-2cea-4538-9cb2-3a5f9fb558cc"). InnerVolumeSpecName "kube-api-access-bhnsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:34.759103 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.759053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:46:34.761383 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.761342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:34.802062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802022    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802059    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802074    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802085    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802097    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802108    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnsj\" (UniqueName: \"kubernetes.io/projected/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-kube-api-access-bhnsj\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:34.802361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:34.802121    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ad718a-2cea-4538-9cb2-3a5f9fb558cc-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:46:35.050091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.050062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:46:35.052770 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:46:35.052741    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0cf199_014f_4541_979d_a848e99e416d.slice/crio-3805bf1a06e2a2f4cdf0e2db6b9baf0ad4e4ac9439dc941708a6f97767309682 WatchSource:0}: Error finding container 3805bf1a06e2a2f4cdf0e2db6b9baf0ad4e4ac9439dc941708a6f97767309682: Status 404 returned error can't find the container with id 3805bf1a06e2a2f4cdf0e2db6b9baf0ad4e4ac9439dc941708a6f97767309682
Sep 10 13:46:35.401162 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.401126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" event={"ID":"4b0cf199-014f-4541-979d-a848e99e416d","Type":"ContainerStarted","Data":"3805bf1a06e2a2f4cdf0e2db6b9baf0ad4e4ac9439dc941708a6f97767309682"}
Sep 10 13:46:35.402891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.402862    2577 generic.go:356] "Generic (PLEG): container finished" podID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerID="bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d" exitCode=0
Sep 10 13:46:35.403042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.402900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerDied","Data":"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"}
Sep 10 13:46:35.403042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.402940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk" event={"ID":"58ad718a-2cea-4538-9cb2-3a5f9fb558cc","Type":"ContainerDied","Data":"0ab58251ba633f4f4cb8831a712d9af48ac4ab172b134a92b8b0be7b50f077d4"}
Sep 10 13:46:35.403042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.402942    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"
Sep 10 13:46:35.403042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.402961    2577 scope.go:117] "RemoveContainer" containerID="bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"
Sep 10 13:46:35.412080 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.412061    2577 scope.go:117] "RemoveContainer" containerID="c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"
Sep 10 13:46:35.420481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.420459    2577 scope.go:117] "RemoveContainer" containerID="bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"
Sep 10 13:46:35.420791 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:35.420774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d\": container with ID starting with bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d not found: ID does not exist" containerID="bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"
Sep 10 13:46:35.420844 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.420801    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d"} err="failed to get container status \"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d\": rpc error: code = NotFound desc = could not find container \"bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d\": container with ID starting with bd19e9c30ae0e69bdff440ddaca57e7cca7705eb188f5e2f1ba851163b6fba4d not found: ID does not exist"
Sep 10 13:46:35.420844 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.420818    2577 scope.go:117] "RemoveContainer" containerID="c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"
Sep 10 13:46:35.421096 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:46:35.421078    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4\": container with ID starting with c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4 not found: ID does not exist" containerID="c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"
Sep 10 13:46:35.421132 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.421103    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4"} err="failed to get container status \"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4\": rpc error: code = NotFound desc = could not find container \"c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4\": container with ID starting with c0037eb390317ba296ab03b2c4acf111655c8350736d7fa0981d1b246d8687f4 not found: ID does not exist"
Sep 10 13:46:35.427848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.427823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:46:35.435733 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:35.435711    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-854ac94d-openshift-default-77c9c96bd4-w7mwk"]
Sep 10 13:46:36.065479 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:36.065442    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" path="/var/lib/kubelet/pods/58ad718a-2cea-4538-9cb2-3a5f9fb558cc/volumes"
Sep 10 13:46:36.408720 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:36.408680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" event={"ID":"4b0cf199-014f-4541-979d-a848e99e416d","Type":"ContainerStarted","Data":"41574b1121c80d5ed0c52f87ae0d1f5ab0a1f9d0b671f7c1924c34fc8aebb303"}
Sep 10 13:46:36.408943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:36.408746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:36.430082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:36.430033    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" podStartSLOduration=2.999109804 podStartE2EDuration="3.430017611s" totalImagesPullingTime="430.907807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:36.429620687 +0000 UTC m=+1080.884495852" watchObservedRunningTime="2026-09-10 13:46:36.430017611 +0000 UTC m=+1080.884892771"
Sep 10 13:46:37.822103 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.822063    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:46:37.822547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.822436    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:37.822547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.822446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:37.822547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.822527    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:37.822547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.822537    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:37.826844 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.826825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.829984 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.829966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-92ef2340-kserve-self-signed-certs\""
Sep 10 13:46:37.888482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.888446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:46:37.931025 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.930989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.931225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.931036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.931225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.931075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.931225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.931117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.931225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.931187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:37.931417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:37.931226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zjn\" (UniqueName: \"kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032739 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032703    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zjn\" (UniqueName: \"kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.032950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.032889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.033226 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.033191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.033268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.033214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.033268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.033241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.035130 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.035103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.035499 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.035478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.042604 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.042581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zjn\" (UniqueName: \"kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn\") pod \"e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") " pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.136185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.136143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:38.412530 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:38.412473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:46:38.415358 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:46:38.415331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a7f5585_283b_4bbd_9a9e_5c34e19191ba.slice/crio-3896c9213713858e482f98e85430ef4b29c9cb820786fa53480e7bddabc08d34 WatchSource:0}: Error finding container 3896c9213713858e482f98e85430ef4b29c9cb820786fa53480e7bddabc08d34: Status 404 returned error can't find the container with id 3896c9213713858e482f98e85430ef4b29c9cb820786fa53480e7bddabc08d34
Sep 10 13:46:39.425994 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:39.425960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" event={"ID":"1a7f5585-283b-4bbd-9a9e-5c34e19191ba","Type":"ContainerStarted","Data":"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"}
Sep 10 13:46:39.426486 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:39.426007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" event={"ID":"1a7f5585-283b-4bbd-9a9e-5c34e19191ba","Type":"ContainerStarted","Data":"3896c9213713858e482f98e85430ef4b29c9cb820786fa53480e7bddabc08d34"}
Sep 10 13:46:39.447000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:39.446943    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" podStartSLOduration=2.446925468 podStartE2EDuration="2.446925468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:39.444055009 +0000 UTC m=+1083.898930184" watchObservedRunningTime="2026-09-10 13:46:39.446925468 +0000 UTC m=+1083.901800630"
Sep 10 13:46:42.420570 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:42.420536    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:46:48.136735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:48.136691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:48.137120 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:48.136761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:48.149416 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:48.149386    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:48.481848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:48.481762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:46:49.749980 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:49.749946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:46:53.236147 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.236104    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:46:53.236750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.236728    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:53.236815 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.236753    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ad718a-2cea-4538-9cb2-3a5f9fb558cc" containerName="istio-proxy"
Sep 10 13:46:53.241459 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.241430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.245927 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.245903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b1f490\"/\"default-dockercfg-d7smr\""
Sep 10 13:46:53.246046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.245908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b1f490\"/\"autores-model-b1f490-kserve-self-signed-certs\""
Sep 10 13:46:53.246361 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.246348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b1f490\"/\"kube-root-ca.crt\""
Sep 10 13:46:53.246407 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.246381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b1f490\"/\"openshift-service-ca.crt\""
Sep 10 13:46:53.252631 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.252603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:46:53.389894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.389854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.389894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.389896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4rd\" (UniqueName: \"kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.390112 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.389930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.390112 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.389962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.390112 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.390011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.390112 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.390084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.490990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.490890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.490989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4rd\" (UniqueName: \"kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491607 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.491607 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.491588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.493439 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.493415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.494014 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.493995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.502321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.502300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4rd\" (UniqueName: \"kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd\") pod \"autores-model-b1f490-kserve-5656695fbc-t8ftk\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") " pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.552977 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.552939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:46:53.826961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:53.826932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:46:53.829322 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:46:53.829294    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb8f23e4_959c_49d7_83c7_1d1c51341427.slice/crio-bd894d81f2dd7bc7013e61f5fc2c786caaa76bdffc532451651fb982ae4dab19 WatchSource:0}: Error finding container bd894d81f2dd7bc7013e61f5fc2c786caaa76bdffc532451651fb982ae4dab19: Status 404 returned error can't find the container with id bd894d81f2dd7bc7013e61f5fc2c786caaa76bdffc532451651fb982ae4dab19
Sep 10 13:46:54.502495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:54.502452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" event={"ID":"db8f23e4-959c-49d7-83c7-1d1c51341427","Type":"ContainerStarted","Data":"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"}
Sep 10 13:46:54.502869 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:54.502527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" event={"ID":"db8f23e4-959c-49d7-83c7-1d1c51341427","Type":"ContainerStarted","Data":"bd894d81f2dd7bc7013e61f5fc2c786caaa76bdffc532451651fb982ae4dab19"}
Sep 10 13:46:54.521497 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:54.521433    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" podStartSLOduration=1.521411369 podStartE2EDuration="1.521411369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:46:54.520427142 +0000 UTC m=+1098.975302305" watchObservedRunningTime="2026-09-10 13:46:54.521411369 +0000 UTC m=+1098.976286533"
Sep 10 13:46:59.254750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:46:59.254715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:47:03.553299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:03.553261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:47:03.553751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:03.553319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:47:03.566390 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:03.566363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:47:04.563540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:04.563498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:47:05.369629 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.369596    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:47:05.369835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.369797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" podUID="4b0cf199-014f-4541-979d-a848e99e416d" containerName="maas-api" containerID="cri-o://41574b1121c80d5ed0c52f87ae0d1f5ab0a1f9d0b671f7c1924c34fc8aebb303" gracePeriod=30
Sep 10 13:47:05.560972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.560934    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b0cf199-014f-4541-979d-a848e99e416d" containerID="41574b1121c80d5ed0c52f87ae0d1f5ab0a1f9d0b671f7c1924c34fc8aebb303" exitCode=0
Sep 10 13:47:05.561176 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.561002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" event={"ID":"4b0cf199-014f-4541-979d-a848e99e416d","Type":"ContainerDied","Data":"41574b1121c80d5ed0c52f87ae0d1f5ab0a1f9d0b671f7c1924c34fc8aebb303"}
Sep 10 13:47:05.615053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.615019    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:47:05.809861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.809767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") pod \"4b0cf199-014f-4541-979d-a848e99e416d\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") "
Sep 10 13:47:05.809861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.809855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95xvj\" (UniqueName: \"kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj\") pod \"4b0cf199-014f-4541-979d-a848e99e416d\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") "
Sep 10 13:47:05.810049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.809894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs\") pod \"4b0cf199-014f-4541-979d-a848e99e416d\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") "
Sep 10 13:47:05.810049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.809923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca\") pod \"4b0cf199-014f-4541-979d-a848e99e416d\" (UID: \"4b0cf199-014f-4541-979d-a848e99e416d\") "
Sep 10 13:47:05.810294 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.810263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b0cf199-014f-4541-979d-a848e99e416d" (UID: "4b0cf199-014f-4541-979d-a848e99e416d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:47:05.811967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.811931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b0cf199-014f-4541-979d-a848e99e416d" (UID: "4b0cf199-014f-4541-979d-a848e99e416d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:05.812378 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.812355    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj" (OuterVolumeSpecName: "kube-api-access-95xvj") pod "4b0cf199-014f-4541-979d-a848e99e416d" (UID: "4b0cf199-014f-4541-979d-a848e99e416d"). InnerVolumeSpecName "kube-api-access-95xvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:05.812474 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.812353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b0cf199-014f-4541-979d-a848e99e416d" (UID: "4b0cf199-014f-4541-979d-a848e99e416d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:05.911029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.910992    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.911029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.911023    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0cf199-014f-4541-979d-a848e99e416d-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.911029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.911033    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b0cf199-014f-4541-979d-a848e99e416d-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.911275 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:05.911041    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xvj\" (UniqueName: \"kubernetes.io/projected/4b0cf199-014f-4541-979d-a848e99e416d-kube-api-access-95xvj\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:06.567950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:06.567921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"
Sep 10 13:47:06.568131 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:06.567957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52" event={"ID":"4b0cf199-014f-4541-979d-a848e99e416d","Type":"ContainerDied","Data":"3805bf1a06e2a2f4cdf0e2db6b9baf0ad4e4ac9439dc941708a6f97767309682"}
Sep 10 13:47:06.568131 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:06.568016    2577 scope.go:117] "RemoveContainer" containerID="41574b1121c80d5ed0c52f87ae0d1f5ab0a1f9d0b671f7c1924c34fc8aebb303"
Sep 10 13:47:06.650299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:06.650265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:47:06.661627 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:06.661592    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0e325272-7d7fff77cf-87b52"]
Sep 10 13:47:08.063970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:08.063932    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0cf199-014f-4541-979d-a848e99e416d" path="/var/lib/kubelet/pods/4b0cf199-014f-4541-979d-a848e99e416d/volumes"
Sep 10 13:47:10.993365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:10.993332    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:10.993780 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:10.993765    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0cf199-014f-4541-979d-a848e99e416d" containerName="maas-api"
Sep 10 13:47:10.993837 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:10.993783    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0cf199-014f-4541-979d-a848e99e416d" containerName="maas-api"
Sep 10 13:47:10.993881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:10.993871    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0cf199-014f-4541-979d-a848e99e416d" containerName="maas-api"
Sep 10 13:47:10.998188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:10.998170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.005874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.005852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-10e9e4e8-serving-cert\""
Sep 10 13:47:11.060730 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.060679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.060915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.060753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.060915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.060878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.060915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.060907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgvp\" (UniqueName: \"kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.063578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.063551    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:11.161789 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.161754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.161986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.161819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.161986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.161840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgvp\" (UniqueName: \"kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.161986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.161882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.162545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.162495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.164269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.164247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.164382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.164360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.175385 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.175352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgvp\" (UniqueName: \"kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp\") pod \"maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.309034 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.308942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:11.608800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:11.608769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:11.610778 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:47:11.610752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b566279_49ea_4cde_aa87_436af6b88c12.slice/crio-6ad25f9444d0e875af44ec3431a229ee4fa508cf643b29accc772c3e5a165894 WatchSource:0}: Error finding container 6ad25f9444d0e875af44ec3431a229ee4fa508cf643b29accc772c3e5a165894: Status 404 returned error can't find the container with id 6ad25f9444d0e875af44ec3431a229ee4fa508cf643b29accc772c3e5a165894
Sep 10 13:47:12.602990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:12.602956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" event={"ID":"9b566279-49ea-4cde-aa87-436af6b88c12","Type":"ContainerStarted","Data":"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"}
Sep 10 13:47:12.603376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:12.603012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:12.603376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:12.603030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" event={"ID":"9b566279-49ea-4cde-aa87-436af6b88c12","Type":"ContainerStarted","Data":"6ad25f9444d0e875af44ec3431a229ee4fa508cf643b29accc772c3e5a165894"}
Sep 10 13:47:12.626845 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:12.626780    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" podStartSLOduration=2.143633208 podStartE2EDuration="2.626760483s" totalImagesPullingTime="483.127275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:12.625591566 +0000 UTC m=+1117.080466729" watchObservedRunningTime="2026-09-10 13:47:12.626760483 +0000 UTC m=+1117.081635655"
Sep 10 13:47:13.571019 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:13.570985    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:47:18.613279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:18.613254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:30.170083 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:30.170047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:47:39.590549 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.590495    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:47:39.591029 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.590721    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" podUID="62e94d02-4393-49a2-b743-8dd9657d69bd" containerName="maas-api" containerID="cri-o://01c13ce4c7006f9423fd74d56c9052728b984e417df0eaf71caef0f039818cde" gracePeriod=30
Sep 10 13:47:39.743153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.743075    2577 generic.go:356] "Generic (PLEG): container finished" podID="62e94d02-4393-49a2-b743-8dd9657d69bd" containerID="01c13ce4c7006f9423fd74d56c9052728b984e417df0eaf71caef0f039818cde" exitCode=0
Sep 10 13:47:39.743322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.743124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" event={"ID":"62e94d02-4393-49a2-b743-8dd9657d69bd","Type":"ContainerDied","Data":"01c13ce4c7006f9423fd74d56c9052728b984e417df0eaf71caef0f039818cde"}
Sep 10 13:47:39.844106 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.844038    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:47:39.959248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.959216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca\") pod \"62e94d02-4393-49a2-b743-8dd9657d69bd\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") "
Sep 10 13:47:39.959463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.959307    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs\") pod \"62e94d02-4393-49a2-b743-8dd9657d69bd\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") "
Sep 10 13:47:39.959463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.959355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf86b\" (UniqueName: \"kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b\") pod \"62e94d02-4393-49a2-b743-8dd9657d69bd\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") "
Sep 10 13:47:39.959463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.959376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") pod \"62e94d02-4393-49a2-b743-8dd9657d69bd\" (UID: \"62e94d02-4393-49a2-b743-8dd9657d69bd\") "
Sep 10 13:47:39.959711 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.959678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62e94d02-4393-49a2-b743-8dd9657d69bd" (UID: "62e94d02-4393-49a2-b743-8dd9657d69bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:47:39.961578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.961545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62e94d02-4393-49a2-b743-8dd9657d69bd" (UID: "62e94d02-4393-49a2-b743-8dd9657d69bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:39.961578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.961570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62e94d02-4393-49a2-b743-8dd9657d69bd" (UID: "62e94d02-4393-49a2-b743-8dd9657d69bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:39.961722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:39.961622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b" (OuterVolumeSpecName: "kube-api-access-qf86b") pod "62e94d02-4393-49a2-b743-8dd9657d69bd" (UID: "62e94d02-4393-49a2-b743-8dd9657d69bd"). InnerVolumeSpecName "kube-api-access-qf86b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:40.060540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.060491    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:40.060540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.060540    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf86b\" (UniqueName: \"kubernetes.io/projected/62e94d02-4393-49a2-b743-8dd9657d69bd-kube-api-access-qf86b\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:40.060540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.060553    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e94d02-4393-49a2-b743-8dd9657d69bd-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:40.060874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.060562    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e94d02-4393-49a2-b743-8dd9657d69bd-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:40.749800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.749769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"
Sep 10 13:47:40.750231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.749809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8" event={"ID":"62e94d02-4393-49a2-b743-8dd9657d69bd","Type":"ContainerDied","Data":"d04f25b484fbcd533c53e822bbda38a8fd2f96d39d1ed115c142124952a42756"}
Sep 10 13:47:40.750231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.749870    2577 scope.go:117] "RemoveContainer" containerID="01c13ce4c7006f9423fd74d56c9052728b984e417df0eaf71caef0f039818cde"
Sep 10 13:47:40.785859 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.785826    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:47:40.787909 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:40.787881    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b1f490-6b74957c77-5vqd8"]
Sep 10 13:47:42.067082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:42.067037    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e94d02-4393-49a2-b743-8dd9657d69bd" path="/var/lib/kubelet/pods/62e94d02-4393-49a2-b743-8dd9657d69bd/volumes"
Sep 10 13:47:42.834299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:42.834264    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:42.834567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:42.834541    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" podUID="9b566279-49ea-4cde-aa87-436af6b88c12" containerName="maas-api" containerID="cri-o://baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775" gracePeriod=30
Sep 10 13:47:43.082702 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.082675    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:43.086137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.086075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls\") pod \"9b566279-49ea-4cde-aa87-436af6b88c12\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") "
Sep 10 13:47:43.086137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.086124    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca\") pod \"9b566279-49ea-4cde-aa87-436af6b88c12\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") "
Sep 10 13:47:43.086293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.086144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs\") pod \"9b566279-49ea-4cde-aa87-436af6b88c12\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") "
Sep 10 13:47:43.086293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.086202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzgvp\" (UniqueName: \"kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp\") pod \"9b566279-49ea-4cde-aa87-436af6b88c12\" (UID: \"9b566279-49ea-4cde-aa87-436af6b88c12\") "
Sep 10 13:47:43.086456 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.086430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b566279-49ea-4cde-aa87-436af6b88c12" (UID: "9b566279-49ea-4cde-aa87-436af6b88c12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:47:43.088336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.088303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp" (OuterVolumeSpecName: "kube-api-access-hzgvp") pod "9b566279-49ea-4cde-aa87-436af6b88c12" (UID: "9b566279-49ea-4cde-aa87-436af6b88c12"). InnerVolumeSpecName "kube-api-access-hzgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:43.088336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.088314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b566279-49ea-4cde-aa87-436af6b88c12" (UID: "9b566279-49ea-4cde-aa87-436af6b88c12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:43.088473 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.088345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b566279-49ea-4cde-aa87-436af6b88c12" (UID: "9b566279-49ea-4cde-aa87-436af6b88c12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:43.187331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.187292    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:43.187331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.187325    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b566279-49ea-4cde-aa87-436af6b88c12-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:43.187331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.187335    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b566279-49ea-4cde-aa87-436af6b88c12-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:43.187599 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.187344    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgvp\" (UniqueName: \"kubernetes.io/projected/9b566279-49ea-4cde-aa87-436af6b88c12-kube-api-access-hzgvp\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:47:43.768785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.768745    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b566279-49ea-4cde-aa87-436af6b88c12" containerID="baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775" exitCode=0
Sep 10 13:47:43.768968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.768784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" event={"ID":"9b566279-49ea-4cde-aa87-436af6b88c12","Type":"ContainerDied","Data":"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"}
Sep 10 13:47:43.768968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.768817    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"
Sep 10 13:47:43.768968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.768896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88" event={"ID":"9b566279-49ea-4cde-aa87-436af6b88c12","Type":"ContainerDied","Data":"6ad25f9444d0e875af44ec3431a229ee4fa508cf643b29accc772c3e5a165894"}
Sep 10 13:47:43.768968 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.768929    2577 scope.go:117] "RemoveContainer" containerID="baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"
Sep 10 13:47:43.778477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.778456    2577 scope.go:117] "RemoveContainer" containerID="baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"
Sep 10 13:47:43.778829 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:47:43.778807    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775\": container with ID starting with baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775 not found: ID does not exist" containerID="baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"
Sep 10 13:47:43.778880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.778844    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775"} err="failed to get container status \"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775\": rpc error: code = NotFound desc = could not find container \"baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775\": container with ID starting with baea60c23220ec7b25938a73d4250e3f0e1af207713733fe28be8b12fb10e775 not found: ID does not exist"
Sep 10 13:47:43.812832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.812802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:43.817783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:43.817758    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10e9e4e8-7c98b7f775-xss88"]
Sep 10 13:47:44.064230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:44.064147    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b566279-49ea-4cde-aa87-436af6b88c12" path="/var/lib/kubelet/pods/9b566279-49ea-4cde-aa87-436af6b88c12/volumes"
Sep 10 13:47:45.088530 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088476    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088893    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e94d02-4393-49a2-b743-8dd9657d69bd" containerName="maas-api"
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088907    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e94d02-4393-49a2-b743-8dd9657d69bd" containerName="maas-api"
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088918    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b566279-49ea-4cde-aa87-436af6b88c12" containerName="maas-api"
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088923    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b566279-49ea-4cde-aa87-436af6b88c12" containerName="maas-api"
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088980    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b566279-49ea-4cde-aa87-436af6b88c12" containerName="maas-api"
Sep 10 13:47:45.089109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.088991    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e94d02-4393-49a2-b743-8dd9657d69bd" containerName="maas-api"
Sep 10 13:47:45.093481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.093459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.097872 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.097853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bc2301a5-openshift-default-dockercfg-b4qcw\""
Sep 10 13:47:45.101932 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.101912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.101957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.102240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.102148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qv9c\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.106705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.106680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:47:45.203279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qv9c\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.203884 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.203820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.204062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.204014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.204187 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.204097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.204187 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.204115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.204305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.204240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.206032 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.206009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.206194 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.206172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.213200 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.213169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qv9c\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.213409 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.213390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token\") pod \"e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") " pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.240999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.240961    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:47:45.404046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.404005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:45.684908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.684814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:47:45.687949 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:47:45.687921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06c0456_5465_46ab_8901_4c4a8970894a.slice/crio-fb1bde767cd0ab3e72209e8c34b5825d8fad52f8d3e6ac10bea7efb813460a9f WatchSource:0}: Error finding container fb1bde767cd0ab3e72209e8c34b5825d8fad52f8d3e6ac10bea7efb813460a9f: Status 404 returned error can't find the container with id fb1bde767cd0ab3e72209e8c34b5825d8fad52f8d3e6ac10bea7efb813460a9f
Sep 10 13:47:45.690228 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.690192    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:47:45.690321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.690285    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:47:45.690321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.690313    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:47:45.783847 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.783809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerStarted","Data":"38deb0e080078bffc3fe97b91b8740642282977906e2fbd2c73a70d461769715"}
Sep 10 13:47:45.784050 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.783875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerStarted","Data":"fb1bde767cd0ab3e72209e8c34b5825d8fad52f8d3e6ac10bea7efb813460a9f"}
Sep 10 13:47:45.821463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:45.821404    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podStartSLOduration=0.821386977 podStartE2EDuration="821.386977ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:45.820905993 +0000 UTC m=+1150.275781156" watchObservedRunningTime="2026-09-10 13:47:45.821386977 +0000 UTC m=+1150.276262140"
Sep 10 13:47:46.405144 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:46.405104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:46.410572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:46.410545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:46.789959 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:46.789868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:46.790719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:46.790698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:47:48.583139 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.583107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:47:48.586903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.586880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.589713 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.589687    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bc2301a5-serving-cert\""
Sep 10 13:47:48.609868 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.609840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:47:48.630428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.630396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddds\" (UniqueName: \"kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.630428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.630436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.630646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.630456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.630646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.630577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.731711 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.731673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.731890 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.731808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddds\" (UniqueName: \"kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.731890 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.731844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.731890 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.731873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.732066 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:47:48.732045    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-serving-cert: secret "maas-api-e2e-derive-bc2301a5-serving-cert" not found
Sep 10 13:47:48.732164 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:47:48.732151    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls podName:bca124fe-9b47-46a1-8eac-e9608bb8e05a nodeName:}" failed. No retries permitted until 2026-09-10 13:47:49.232118453 +0000 UTC m=+1153.686993617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls") pod "maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" (UID: "bca124fe-9b47-46a1-8eac-e9608bb8e05a") : secret "maas-api-e2e-derive-bc2301a5-serving-cert" not found
Sep 10 13:47:48.732462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.732441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.734142 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.734113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.747418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.747389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddds\" (UniqueName: \"kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:48.944344 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:48.944311    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:47:49.236306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:49.236202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:49.238732 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:49.238704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") pod \"maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:49.498618 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:49.498535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:49.779558 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:49.779529    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:47:49.781417 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:47:49.781390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca124fe_9b47_46a1_8eac_e9608bb8e05a.slice/crio-0da7c427421fc23a66dcc2f3b860c534decf360d66cf62ccb440c9b2ea9031e8 WatchSource:0}: Error finding container 0da7c427421fc23a66dcc2f3b860c534decf360d66cf62ccb440c9b2ea9031e8: Status 404 returned error can't find the container with id 0da7c427421fc23a66dcc2f3b860c534decf360d66cf62ccb440c9b2ea9031e8
Sep 10 13:47:49.806893 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:49.806858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" event={"ID":"bca124fe-9b47-46a1-8eac-e9608bb8e05a","Type":"ContainerStarted","Data":"0da7c427421fc23a66dcc2f3b860c534decf360d66cf62ccb440c9b2ea9031e8"}
Sep 10 13:47:50.813586 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:50.813548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" event={"ID":"bca124fe-9b47-46a1-8eac-e9608bb8e05a","Type":"ContainerStarted","Data":"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"}
Sep 10 13:47:50.814037 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:50.813633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:47:50.848558 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:50.848470    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" podStartSLOduration=2.314936658 podStartE2EDuration="2.848447531s" totalImagesPullingTime="533.510873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:50.844736722 +0000 UTC m=+1155.299611884" watchObservedRunningTime="2026-09-10 13:47:50.848447531 +0000 UTC m=+1155.303322697"
Sep 10 13:47:56.824687 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:47:56.824654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:48:00.261240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.261119    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" podUID="db8f23e4-959c-49d7-83c7-1d1c51341427" containerName="main" containerID="cri-o://0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368" gracePeriod=45
Sep 10 13:48:00.503119 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.503087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.551853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.551925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.551969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.552093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.552153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn4rd\" (UniqueName: \"kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.552805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.552184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home\") pod \"db8f23e4-959c-49d7-83c7-1d1c51341427\" (UID: \"db8f23e4-959c-49d7-83c7-1d1c51341427\") "
Sep 10 13:48:00.553776 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.553390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home" (OuterVolumeSpecName: "home") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:00.556264 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.555715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:00.562601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.558596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache" (OuterVolumeSpecName: "model-cache") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:00.566719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.566662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:00.571032 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.570982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm" (OuterVolumeSpecName: "dshm") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:00.571625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.571587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd" (OuterVolumeSpecName: "kube-api-access-mn4rd") pod "db8f23e4-959c-49d7-83c7-1d1c51341427" (UID: "db8f23e4-959c-49d7-83c7-1d1c51341427"). InnerVolumeSpecName "kube-api-access-mn4rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655167    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655213    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn4rd\" (UniqueName: \"kubernetes.io/projected/db8f23e4-959c-49d7-83c7-1d1c51341427-kube-api-access-mn4rd\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655231    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655247    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655261    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db8f23e4-959c-49d7-83c7-1d1c51341427-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.655309 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.655276    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db8f23e4-959c-49d7-83c7-1d1c51341427-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:00.872204 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.872166    2577 generic.go:356] "Generic (PLEG): container finished" podID="db8f23e4-959c-49d7-83c7-1d1c51341427" containerID="0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368" exitCode=0
Sep 10 13:48:00.872204 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.872197    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"
Sep 10 13:48:00.872495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.872216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" event={"ID":"db8f23e4-959c-49d7-83c7-1d1c51341427","Type":"ContainerDied","Data":"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"}
Sep 10 13:48:00.872495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.872274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk" event={"ID":"db8f23e4-959c-49d7-83c7-1d1c51341427","Type":"ContainerDied","Data":"bd894d81f2dd7bc7013e61f5fc2c786caaa76bdffc532451651fb982ae4dab19"}
Sep 10 13:48:00.872495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.872297    2577 scope.go:117] "RemoveContainer" containerID="0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"
Sep 10 13:48:00.884964 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.884938    2577 scope.go:117] "RemoveContainer" containerID="0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"
Sep 10 13:48:00.885318 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:48:00.885294    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368\": container with ID starting with 0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368 not found: ID does not exist" containerID="0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"
Sep 10 13:48:00.885434 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.885327    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368"} err="failed to get container status \"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368\": rpc error: code = NotFound desc = could not find container \"0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368\": container with ID starting with 0dafb3006327f0efc2235c1cbfccf92a11606de7d16e356294396ada9303c368 not found: ID does not exist"
Sep 10 13:48:00.908064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.908022    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:48:00.916319 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:00.916287    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b1f490/autores-model-b1f490-kserve-5656695fbc-t8ftk"]
Sep 10 13:48:02.067482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:02.067447    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8f23e4-959c-49d7-83c7-1d1c51341427" path="/var/lib/kubelet/pods/db8f23e4-959c-49d7-83c7-1d1c51341427/volumes"
Sep 10 13:48:07.543836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:07.543785    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:48:09.929617 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.929584    2577 generic.go:356] "Generic (PLEG): container finished" podID="f06c0456-5465-46ab-8901-4c4a8970894a" containerID="38deb0e080078bffc3fe97b91b8740642282977906e2fbd2c73a70d461769715" exitCode=0
Sep 10 13:48:09.930092 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.929630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerDied","Data":"38deb0e080078bffc3fe97b91b8740642282977906e2fbd2c73a70d461769715"}
Sep 10 13:48:09.930092 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.930021    2577 scope.go:117] "RemoveContainer" containerID="38deb0e080078bffc3fe97b91b8740642282977906e2fbd2c73a70d461769715"
Sep 10 13:48:09.931129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.931092    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:09.931238 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.931185    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:09.931302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:09.931238    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:10.939425 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:10.938333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerStarted","Data":"b55001b53edf7412ccd9a264d0548978d1862965cd0c2bb7fd8128dce0f211a9"}
Sep 10 13:48:11.404166 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:11.404127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:11.404383 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:11.404348    2577 patch_prober.go:28] interesting pod/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 13:48:11.404462 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:11.404409    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 13:48:12.404661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:12.404625    2577 patch_prober.go:28] interesting pod/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 13:48:12.405094 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:12.404696    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 13:48:13.405048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:13.405012    2577 patch_prober.go:28] interesting pod/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 13:48:13.405452 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:13.405087    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 13:48:14.404542 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:14.404488    2577 patch_prober.go:28] interesting pod/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 13:48:14.404725 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:14.404564    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 13:48:15.405006 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:15.404970    2577 patch_prober.go:28] interesting pod/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 10 13:48:15.405422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:15.405044    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 10 13:48:16.408967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:16.408934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:16.409413 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:16.409189    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:16.410097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:16.410080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:19.376793 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.376756    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:48:19.377227 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.376960    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" podUID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" containerName="maas-api" containerID="cri-o://eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4" gracePeriod=30
Sep 10 13:48:19.637775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.637698    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:48:19.754049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.754009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lddds\" (UniqueName: \"kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds\") pod \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") "
Sep 10 13:48:19.754248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.754105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca\") pod \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") "
Sep 10 13:48:19.754248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.754161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs\") pod \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") "
Sep 10 13:48:19.754248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.754191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") pod \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\" (UID: \"bca124fe-9b47-46a1-8eac-e9608bb8e05a\") "
Sep 10 13:48:19.754749 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.754701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bca124fe-9b47-46a1-8eac-e9608bb8e05a" (UID: "bca124fe-9b47-46a1-8eac-e9608bb8e05a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:48:19.756603 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.756573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds" (OuterVolumeSpecName: "kube-api-access-lddds") pod "bca124fe-9b47-46a1-8eac-e9608bb8e05a" (UID: "bca124fe-9b47-46a1-8eac-e9608bb8e05a"). InnerVolumeSpecName "kube-api-access-lddds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:19.756718 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.756682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bca124fe-9b47-46a1-8eac-e9608bb8e05a" (UID: "bca124fe-9b47-46a1-8eac-e9608bb8e05a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:19.756797 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.756729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bca124fe-9b47-46a1-8eac-e9608bb8e05a" (UID: "bca124fe-9b47-46a1-8eac-e9608bb8e05a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:19.833726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.833689    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:48:19.833942 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.833914    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy" containerID="cri-o://b55001b53edf7412ccd9a264d0548978d1862965cd0c2bb7fd8128dce0f211a9" gracePeriod=30
Sep 10 13:48:19.856072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.856020    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:19.856072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.856064    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bca124fe-9b47-46a1-8eac-e9608bb8e05a-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:19.856072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.856074    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddds\" (UniqueName: \"kubernetes.io/projected/bca124fe-9b47-46a1-8eac-e9608bb8e05a-kube-api-access-lddds\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:19.856072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.856084    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bca124fe-9b47-46a1-8eac-e9608bb8e05a-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:19.994463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.994361    2577 generic.go:356] "Generic (PLEG): container finished" podID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" containerID="eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4" exitCode=0
Sep 10 13:48:19.994463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.994411    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"
Sep 10 13:48:19.994463 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.994449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" event={"ID":"bca124fe-9b47-46a1-8eac-e9608bb8e05a","Type":"ContainerDied","Data":"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"}
Sep 10 13:48:19.994748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.994523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79" event={"ID":"bca124fe-9b47-46a1-8eac-e9608bb8e05a","Type":"ContainerDied","Data":"0da7c427421fc23a66dcc2f3b860c534decf360d66cf62ccb440c9b2ea9031e8"}
Sep 10 13:48:19.994748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:19.994546    2577 scope.go:117] "RemoveContainer" containerID="eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"
Sep 10 13:48:20.004377 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:20.004357    2577 scope.go:117] "RemoveContainer" containerID="eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"
Sep 10 13:48:20.004696 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:48:20.004678    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4\": container with ID starting with eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4 not found: ID does not exist" containerID="eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"
Sep 10 13:48:20.004754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:20.004705    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4"} err="failed to get container status \"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4\": rpc error: code = NotFound desc = could not find container \"eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4\": container with ID starting with eb71fba0eff48938c026cab1e226e38ea3098db005748968159d6d11b8b545e4 not found: ID does not exist"
Sep 10 13:48:20.017995 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:20.017959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:48:20.024034 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:20.024010    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bc2301a5-856bfbf494-ntq79"]
Sep 10 13:48:20.064175 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:20.064139    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" path="/var/lib/kubelet/pods/bca124fe-9b47-46a1-8eac-e9608bb8e05a/volumes"
Sep 10 13:48:25.026364 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.026325    2577 generic.go:356] "Generic (PLEG): container finished" podID="f06c0456-5465-46ab-8901-4c4a8970894a" containerID="b55001b53edf7412ccd9a264d0548978d1862965cd0c2bb7fd8128dce0f211a9" exitCode=0
Sep 10 13:48:25.026788 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.026390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerDied","Data":"b55001b53edf7412ccd9a264d0548978d1862965cd0c2bb7fd8128dce0f211a9"}
Sep 10 13:48:25.026788 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.026455    2577 scope.go:117] "RemoveContainer" containerID="38deb0e080078bffc3fe97b91b8740642282977906e2fbd2c73a70d461769715"
Sep 10 13:48:25.090265 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.090230    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:25.096099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096157    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096183    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qv9c\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096256    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096294    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096301    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:25.096545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token\") pod \"f06c0456-5465-46ab-8901-4c4a8970894a\" (UID: \"f06c0456-5465-46ab-8901-4c4a8970894a\") "
Sep 10 13:48:25.096545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:25.096545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096776    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096789    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096801    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096815    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.096817 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.096827    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06c0456-5465-46ab-8901-4c4a8970894a-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.098265 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.098242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:25.098481 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.098468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:25.098709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.098685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:48:25.098709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.098701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c" (OuterVolumeSpecName: "kube-api-access-9qv9c") pod "f06c0456-5465-46ab-8901-4c4a8970894a" (UID: "f06c0456-5465-46ab-8901-4c4a8970894a"). InnerVolumeSpecName "kube-api-access-9qv9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:25.197933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.197892    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.197933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.197924    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06c0456-5465-46ab-8901-4c4a8970894a-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.197933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.197937    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06c0456-5465-46ab-8901-4c4a8970894a-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.198168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.197948    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qv9c\" (UniqueName: \"kubernetes.io/projected/f06c0456-5465-46ab-8901-4c4a8970894a-kube-api-access-9qv9c\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:25.291296 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:48:25.291675 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291661    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" containerName="maas-api"
Sep 10 13:48:25.291726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291677    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" containerName="maas-api"
Sep 10 13:48:25.291726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291685    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:25.291726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291690    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:25.291726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291709    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8f23e4-959c-49d7-83c7-1d1c51341427" containerName="main"
Sep 10 13:48:25.291726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291715    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8f23e4-959c-49d7-83c7-1d1c51341427" containerName="main"
Sep 10 13:48:25.291880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291779    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8f23e4-959c-49d7-83c7-1d1c51341427" containerName="main"
Sep 10 13:48:25.291880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291788    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:25.291880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca124fe-9b47-46a1-8eac-e9608bb8e05a" containerName="maas-api"
Sep 10 13:48:25.291880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.291800    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:25.295189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.295173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5d0c07-openshift-default-dockercfg-4dldd\""
Sep 10 13:48:25.298405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298480 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2zq\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298480 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298616 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298616 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298616 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.298779 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.298708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.315329 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.315300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:48:25.399931 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.399894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.399962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2zq\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400427 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400300    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400486 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400807 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.400860 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.400824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.402297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.402280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.402735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.402718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.412027 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.411995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2zq\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.412775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.412754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token\") pod \"e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") " pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.605391 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.605352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:25.887499 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.887474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:48:25.889569 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:48:25.889541    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02dd2157_7b62_478e_b39f_e450ed21b383.slice/crio-790a8b3b05faf66c64ea2f9157f1238fdbb9d68001db5b102ba193e975ce4fb0 WatchSource:0}: Error finding container 790a8b3b05faf66c64ea2f9157f1238fdbb9d68001db5b102ba193e975ce4fb0: Status 404 returned error can't find the container with id 790a8b3b05faf66c64ea2f9157f1238fdbb9d68001db5b102ba193e975ce4fb0
Sep 10 13:48:25.891865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.891822    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:25.891946 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.891928    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:25.891999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:25.891984    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:26.033439 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.033404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"
Sep 10 13:48:26.033933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.033410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8" event={"ID":"f06c0456-5465-46ab-8901-4c4a8970894a","Type":"ContainerDied","Data":"fb1bde767cd0ab3e72209e8c34b5825d8fad52f8d3e6ac10bea7efb813460a9f"}
Sep 10 13:48:26.033933 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.033568    2577 scope.go:117] "RemoveContainer" containerID="b55001b53edf7412ccd9a264d0548978d1862965cd0c2bb7fd8128dce0f211a9"
Sep 10 13:48:26.035292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.035257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerStarted","Data":"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"}
Sep 10 13:48:26.035403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.035316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerStarted","Data":"790a8b3b05faf66c64ea2f9157f1238fdbb9d68001db5b102ba193e975ce4fb0"}
Sep 10 13:48:26.096493 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.096417    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podStartSLOduration=1.096396297 podStartE2EDuration="1.096396297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:26.094692492 +0000 UTC m=+1190.549567657" watchObservedRunningTime="2026-09-10 13:48:26.096396297 +0000 UTC m=+1190.551271460"
Sep 10 13:48:26.153958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.153919    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:48:26.163970 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.163896    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bc2301a5-openshift-default-66845f57cb-fpcd8"]
Sep 10 13:48:26.605874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.605761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:26.611274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:26.611237    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:27.042714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:27.042686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:27.043746 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:27.043727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:28.063757 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:28.063722    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" path="/var/lib/kubelet/pods/f06c0456-5465-46ab-8901-4c4a8970894a/volumes"
Sep 10 13:48:28.176644 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:28.176604    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:48:29.363808 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.363761    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:48:29.364385 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.364361    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:29.364469 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.364389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f06c0456-5465-46ab-8901-4c4a8970894a" containerName="istio-proxy"
Sep 10 13:48:29.368945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.368921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.371827 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.371772    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5d0c07-serving-cert\""
Sep 10 13:48:29.385329 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.385296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:48:29.437851 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.437814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.438027 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.437964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czzp\" (UniqueName: \"kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.438027 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.438016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.438152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.438062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.538587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.538551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czzp\" (UniqueName: \"kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.538763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.538604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.538763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.538644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.538763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.538735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.539319 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.539294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.541160 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.541131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.541475 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.541452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.557116 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.557090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czzp\" (UniqueName: \"kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp\") pod \"maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.681572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.681454    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:29.963841 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.963807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:48:29.965920 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:48:29.965884    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e254439_dcbf_43e1_ab72_e3a6e29e2355.slice/crio-d8e4a78d9e54d46e74f2e8b91f5e8a1566bdddead8ba1fe2945c080d0e4d55f9 WatchSource:0}: Error finding container d8e4a78d9e54d46e74f2e8b91f5e8a1566bdddead8ba1fe2945c080d0e4d55f9: Status 404 returned error can't find the container with id d8e4a78d9e54d46e74f2e8b91f5e8a1566bdddead8ba1fe2945c080d0e4d55f9
Sep 10 13:48:29.967637 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:29.967615    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:48:30.068062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:30.068026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" event={"ID":"8e254439-dcbf-43e1-ab72-e3a6e29e2355","Type":"ContainerStarted","Data":"d8e4a78d9e54d46e74f2e8b91f5e8a1566bdddead8ba1fe2945c080d0e4d55f9"}
Sep 10 13:48:31.071654 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:31.071602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" event={"ID":"8e254439-dcbf-43e1-ab72-e3a6e29e2355","Type":"ContainerStarted","Data":"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"}
Sep 10 13:48:31.072106 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:31.071672    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:31.094031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:31.093981    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" podStartSLOduration=1.5727732319999999 podStartE2EDuration="2.093965241s" totalImagesPullingTime="521.192009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:31.092015013 +0000 UTC m=+1195.546890191" watchObservedRunningTime="2026-09-10 13:48:31.093965241 +0000 UTC m=+1195.548840403"
Sep 10 13:48:36.076572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:36.076536    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:48:36.078440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:36.078419    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:48:36.804520 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:36.804476    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:48:37.082223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:37.082148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:48:44.539550 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:44.539496    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:48:47.152571 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.152492    2577 generic.go:356] "Generic (PLEG): container finished" podID="02dd2157-7b62-478e-b39f-e450ed21b383" containerID="e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf" exitCode=0
Sep 10 13:48:47.153007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.152572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerDied","Data":"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"}
Sep 10 13:48:47.153055 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.153039    2577 scope.go:117] "RemoveContainer" containerID="e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"
Sep 10 13:48:47.154214 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.154186    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:47.154281 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.154262    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:47.154318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:47.154297    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:48:48.164099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:48.164063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerStarted","Data":"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"}
Sep 10 13:48:48.606175 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:48.606074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:48.606399 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:48.606371    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 13:48:48.606486 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:48.606429    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 13:48:49.605967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:49.605923    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 13:48:49.606371 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:49.605992    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 13:48:50.606249 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:50.606207    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 13:48:50.606687 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:50.606271    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 13:48:51.606254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:51.606214    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 13:48:51.606681 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:51.606290    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 13:48:51.824025 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:51.823961    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" podUID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" containerName="main" containerID="cri-o://7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da" gracePeriod=45
Sep 10 13:48:52.117265 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.117240    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:48:52.185170 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.185130    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" containerID="7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da" exitCode=0
Sep 10 13:48:52.185348 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.185174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"
Sep 10 13:48:52.185348 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.185206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" event={"ID":"1a7f5585-283b-4bbd-9a9e-5c34e19191ba","Type":"ContainerDied","Data":"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"}
Sep 10 13:48:52.185348 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.185250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d" event={"ID":"1a7f5585-283b-4bbd-9a9e-5c34e19191ba","Type":"ContainerDied","Data":"3896c9213713858e482f98e85430ef4b29c9cb820786fa53480e7bddabc08d34"}
Sep 10 13:48:52.185348 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.185270    2577 scope.go:117] "RemoveContainer" containerID="7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"
Sep 10 13:48:52.193843 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.193823    2577 scope.go:117] "RemoveContainer" containerID="7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"
Sep 10 13:48:52.194126 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:48:52.194110    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da\": container with ID starting with 7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da not found: ID does not exist" containerID="7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"
Sep 10 13:48:52.194178 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.194134    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da"} err="failed to get container status \"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da\": rpc error: code = NotFound desc = could not find container \"7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da\": container with ID starting with 7b4551f483c7e0aae93b93b64e12dcd722eadae6105499e02df4bdc32b3921da not found: ID does not exist"
Sep 10 13:48:52.286740 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.286925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.286925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8zjn\" (UniqueName: \"kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.286925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.286925 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.287096 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.286964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir\") pod \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\" (UID: \"1a7f5585-283b-4bbd-9a9e-5c34e19191ba\") "
Sep 10 13:48:52.287096 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.287039    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:52.287191 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.287122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home" (OuterVolumeSpecName: "home") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:52.287244 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.287206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:52.287301 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.287268    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.287301 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.287289    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.289157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.289127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm" (OuterVolumeSpecName: "dshm") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:48:52.289289 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.289174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn" (OuterVolumeSpecName: "kube-api-access-l8zjn") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "kube-api-access-l8zjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:48:52.289289 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.289177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a7f5585-283b-4bbd-9a9e-5c34e19191ba" (UID: "1a7f5585-283b-4bbd-9a9e-5c34e19191ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:48:52.388610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.388569    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.388610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.388600    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.388610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.388610    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zjn\" (UniqueName: \"kubernetes.io/projected/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-kube-api-access-l8zjn\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.388610 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.388620    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7f5585-283b-4bbd-9a9e-5c34e19191ba-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:48:52.512313 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.512279    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:48:52.515287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.515262    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-92ef2340-kserve-5c5495995b-pjq8d"]
Sep 10 13:48:52.606218 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.606177    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 10 13:48:52.606398 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:52.606239    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 10 13:48:53.610161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:53.610131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:53.610597 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:53.610414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:53.611078 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:53.611058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:48:54.064327 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:54.064291    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" path="/var/lib/kubelet/pods/1a7f5585-283b-4bbd-9a9e-5c34e19191ba/volumes"
Sep 10 13:48:56.530225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.530180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:48:56.530823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.530802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" containerName="main"
Sep 10 13:48:56.530908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.530826    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" containerName="main"
Sep 10 13:48:56.530966 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.530950    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7f5585-283b-4bbd-9a9e-5c34e19191ba" containerName="main"
Sep 10 13:48:56.533368 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.533347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.535924 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.535897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02fc2e\"/\"kube-root-ca.crt\""
Sep 10 13:48:56.536796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.536776    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02fc2e\"/\"openshift-service-ca.crt\""
Sep 10 13:48:56.536914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.536796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02fc2e\"/\"default-dockercfg-pk5mv\""
Sep 10 13:48:56.536914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.536777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02fc2e\"/\"test-model-02fc2e-kserve-self-signed-certs\""
Sep 10 13:48:56.555428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.555401    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:48:56.625832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.625801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.626012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.625837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.626012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.625858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.626012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.625883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.626012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.625959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lzwj\" (UniqueName: \"kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.626012 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.626004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726641 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726641 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lzwj\" (UniqueName: \"kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.726865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.726778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.727090 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.727062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.727138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.727107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.727212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.727194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.729046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.729022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.729176 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.729160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.763944 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.763910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lzwj\" (UniqueName: \"kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj\") pod \"test-model-02fc2e-kserve-85df95fb5-7vnd8\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") " pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:56.843595 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:56.843519    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:48:57.127709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:57.127675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:48:57.130839 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:48:57.130815    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7de09390_66d4_4174_b6e1_f507c345b174.slice/crio-e020d94fcf3cbd2c2cb635de06e821964a9a136c776787c8b0fc39a038e72f96 WatchSource:0}: Error finding container e020d94fcf3cbd2c2cb635de06e821964a9a136c776787c8b0fc39a038e72f96: Status 404 returned error can't find the container with id e020d94fcf3cbd2c2cb635de06e821964a9a136c776787c8b0fc39a038e72f96
Sep 10 13:48:57.214253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:57.214217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" event={"ID":"7de09390-66d4-4174-b6e1-f507c345b174","Type":"ContainerStarted","Data":"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"}
Sep 10 13:48:57.214465 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:57.214265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" event={"ID":"7de09390-66d4-4174-b6e1-f507c345b174","Type":"ContainerStarted","Data":"e020d94fcf3cbd2c2cb635de06e821964a9a136c776787c8b0fc39a038e72f96"}
Sep 10 13:48:57.239421 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:48:57.239354    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" podStartSLOduration=1.239332473 podStartE2EDuration="1.239332473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:57.237103917 +0000 UTC m=+1221.691979082" watchObservedRunningTime="2026-09-10 13:48:57.239332473 +0000 UTC m=+1221.694207638"
Sep 10 13:49:02.081708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:02.081677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:49:06.843873 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:06.843844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:49:06.844315 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:06.843891    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:49:06.857064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:06.857037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:49:07.276951 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:07.276919    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:49:17.341378 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.341342    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:17.343823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.343802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.349850 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.349814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d0c07\"/\"kube-root-ca.crt\""
Sep 10 13:49:17.350000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.349884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5d0c07\"/\"openshift-service-ca.crt\""
Sep 10 13:49:17.350000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.349939    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d0c07\"/\"default-dockercfg-d9jkr\""
Sep 10 13:49:17.350000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.349884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5d0c07\"/\"explicit-model-5d0c07-kserve-self-signed-certs\""
Sep 10 13:49:17.365392 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.365362    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:17.419496 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.419700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljrn\" (UniqueName: \"kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.419700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.419700 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.419813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.419813 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.419764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.520765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.520724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.520965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.520779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljrn\" (UniqueName: \"kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.520965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.520819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.520965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.520870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.521162 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.521048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.521162 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.521091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.521276 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.521209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.521276 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.521217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.521365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.521347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.523199 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.523166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.523346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.523332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.543252 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.543218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljrn\" (UniqueName: \"kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn\") pod \"explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") " pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.656778 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.656730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:17.931428 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:17.931400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:17.933237 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:49:17.933207    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82aecc3d_12e3_45c3_bf51_6e04581b82f8.slice/crio-5c75143ce899e6150bd6dcb4d810fc1b3271d0de947b9ce30a12187ee5958f6b WatchSource:0}: Error finding container 5c75143ce899e6150bd6dcb4d810fc1b3271d0de947b9ce30a12187ee5958f6b: Status 404 returned error can't find the container with id 5c75143ce899e6150bd6dcb4d810fc1b3271d0de947b9ce30a12187ee5958f6b
Sep 10 13:49:18.320567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:18.320434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" event={"ID":"82aecc3d-12e3-45c3-bf51-6e04581b82f8","Type":"ContainerStarted","Data":"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"}
Sep 10 13:49:18.320567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:18.320487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" event={"ID":"82aecc3d-12e3-45c3-bf51-6e04581b82f8","Type":"ContainerStarted","Data":"5c75143ce899e6150bd6dcb4d810fc1b3271d0de947b9ce30a12187ee5958f6b"}
Sep 10 13:49:18.340794 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:18.340737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" podStartSLOduration=1.340720401 podStartE2EDuration="1.340720401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:49:18.339729359 +0000 UTC m=+1242.794604524" watchObservedRunningTime="2026-09-10 13:49:18.340720401 +0000 UTC m=+1242.795595572"
Sep 10 13:49:20.653248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:20.653213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:49:27.657586 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:27.657539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:27.658089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:27.657605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:27.670572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:27.670542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:28.383581 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:28.383549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:30.120714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:30.120680    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:37.162138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:37.162103    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:49:41.623546 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.623485    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:49:41.624091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.623757    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" podUID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" containerName="maas-api" containerID="cri-o://17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0" gracePeriod=30
Sep 10 13:49:41.901915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.901878    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:49:41.964187 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.964155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4czzp\" (UniqueName: \"kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp\") pod \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") "
Sep 10 13:49:41.964367 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.964199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls\") pod \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") "
Sep 10 13:49:41.964367 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.964257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs\") pod \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") "
Sep 10 13:49:41.964367 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.964329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca\") pod \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\" (UID: \"8e254439-dcbf-43e1-ab72-e3a6e29e2355\") "
Sep 10 13:49:41.964977 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.964945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8e254439-dcbf-43e1-ab72-e3a6e29e2355" (UID: "8e254439-dcbf-43e1-ab72-e3a6e29e2355"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:41.966539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.966483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8e254439-dcbf-43e1-ab72-e3a6e29e2355" (UID: "8e254439-dcbf-43e1-ab72-e3a6e29e2355"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:41.966539 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.966502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8e254439-dcbf-43e1-ab72-e3a6e29e2355" (UID: "8e254439-dcbf-43e1-ab72-e3a6e29e2355"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:41.966750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:41.966731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp" (OuterVolumeSpecName: "kube-api-access-4czzp") pod "8e254439-dcbf-43e1-ab72-e3a6e29e2355" (UID: "8e254439-dcbf-43e1-ab72-e3a6e29e2355"). InnerVolumeSpecName "kube-api-access-4czzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:42.065181 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.065145    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e254439-dcbf-43e1-ab72-e3a6e29e2355-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:42.065181 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.065175    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czzp\" (UniqueName: \"kubernetes.io/projected/8e254439-dcbf-43e1-ab72-e3a6e29e2355-kube-api-access-4czzp\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:42.065181 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.065190    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:42.065489 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.065199    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e254439-dcbf-43e1-ab72-e3a6e29e2355-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:42.445701 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.445668    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" containerID="17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0" exitCode=0
Sep 10 13:49:42.445880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.445717    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"
Sep 10 13:49:42.445880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.445742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" event={"ID":"8e254439-dcbf-43e1-ab72-e3a6e29e2355","Type":"ContainerDied","Data":"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"}
Sep 10 13:49:42.445880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.445785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf" event={"ID":"8e254439-dcbf-43e1-ab72-e3a6e29e2355","Type":"ContainerDied","Data":"d8e4a78d9e54d46e74f2e8b91f5e8a1566bdddead8ba1fe2945c080d0e4d55f9"}
Sep 10 13:49:42.445880 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.445808    2577 scope.go:117] "RemoveContainer" containerID="17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"
Sep 10 13:49:42.454262 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.454243    2577 scope.go:117] "RemoveContainer" containerID="17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"
Sep 10 13:49:42.454565 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:49:42.454541    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0\": container with ID starting with 17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0 not found: ID does not exist" containerID="17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"
Sep 10 13:49:42.454617 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.454575    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0"} err="failed to get container status \"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0\": rpc error: code = NotFound desc = could not find container \"17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0\": container with ID starting with 17bfdd7d44398077ad93e8b47ed3bdc7b14c25deb00342467d4b260fa907b9c0 not found: ID does not exist"
Sep 10 13:49:42.477834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.477805    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:49:42.485729 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:42.485693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5d0c07-7df65ff7b5-stvkf"]
Sep 10 13:49:44.064086 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:44.064050    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" path="/var/lib/kubelet/pods/8e254439-dcbf-43e1-ab72-e3a6e29e2355/volumes"
Sep 10 13:49:45.402523 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.402469    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" podUID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" containerName="main" containerID="cri-o://b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3" gracePeriod=45
Sep 10 13:49:45.654248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.654178    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:45.699587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.699743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.699743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.699849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.699849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.699849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zljrn\" (UniqueName: \"kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn\") pod \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\" (UID: \"82aecc3d-12e3-45c3-bf51-6e04581b82f8\") "
Sep 10 13:49:45.700018 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.699935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:45.700082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.700016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home" (OuterVolumeSpecName: "home") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:45.700082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.700050    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:45.700250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.700221    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:45.700250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.700251    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:45.700437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.700266    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:45.701999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.701968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm" (OuterVolumeSpecName: "dshm") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:45.702098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.702001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn" (OuterVolumeSpecName: "kube-api-access-zljrn") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "kube-api-access-zljrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:45.702098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.702044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82aecc3d-12e3-45c3-bf51-6e04581b82f8" (UID: "82aecc3d-12e3-45c3-bf51-6e04581b82f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:45.800867 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.800823    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82aecc3d-12e3-45c3-bf51-6e04581b82f8-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:45.800867 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.800859    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82aecc3d-12e3-45c3-bf51-6e04581b82f8-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:45.800867 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:45.800871    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljrn\" (UniqueName: \"kubernetes.io/projected/82aecc3d-12e3-45c3-bf51-6e04581b82f8-kube-api-access-zljrn\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:46.466078 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.466043    2577 generic.go:356] "Generic (PLEG): container finished" podID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" containerID="b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3" exitCode=0
Sep 10 13:49:46.466543 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.466088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"
Sep 10 13:49:46.466543 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.466125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" event={"ID":"82aecc3d-12e3-45c3-bf51-6e04581b82f8","Type":"ContainerDied","Data":"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"}
Sep 10 13:49:46.466543 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.466168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd" event={"ID":"82aecc3d-12e3-45c3-bf51-6e04581b82f8","Type":"ContainerDied","Data":"5c75143ce899e6150bd6dcb4d810fc1b3271d0de947b9ce30a12187ee5958f6b"}
Sep 10 13:49:46.466543 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.466184    2577 scope.go:117] "RemoveContainer" containerID="b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"
Sep 10 13:49:46.474496 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.474478    2577 scope.go:117] "RemoveContainer" containerID="b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"
Sep 10 13:49:46.474812 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:49:46.474791    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3\": container with ID starting with b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3 not found: ID does not exist" containerID="b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"
Sep 10 13:49:46.474857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.474821    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3"} err="failed to get container status \"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3\": rpc error: code = NotFound desc = could not find container \"b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3\": container with ID starting with b6de8ba7f9ac53e4caacc1eb0ab09749f523a68bc6a9aafafe119f9257d40bf3 not found: ID does not exist"
Sep 10 13:49:46.489049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.489019    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:46.496030 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:46.496001    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5d0c07/explicit-model-5d0c07-kserve-84dd86f5ff-bt4gd"]
Sep 10 13:49:48.064381 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:48.064343    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" path="/var/lib/kubelet/pods/82aecc3d-12e3-45c3-bf51-6e04581b82f8/volumes"
Sep 10 13:49:53.181756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:53.181723    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:49:53.182229 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:53.181943    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" containerID="cri-o://eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a" gracePeriod=30
Sep 10 13:49:53.612981 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:53.612899    2577 patch_prober.go:28] interesting pod/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:49:53.612981 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:53.612970    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:49:56.158985 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:56.158942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:49:58.440247 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.440217    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:49:58.526332 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526242    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526332 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526332 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526392    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt2zq\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526473    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data\") pod \"02dd2157-7b62-478e-b39f-e450ed21b383\" (UID: \"02dd2157-7b62-478e-b39f-e450ed21b383\") "
Sep 10 13:49:58.526956 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:58.526956 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:58.526956 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data" (OuterVolumeSpecName: "istio-data") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:58.527109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.526954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:58.527109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.527013    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.527109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.527035    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02dd2157-7b62-478e-b39f-e450ed21b383-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.527260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.527110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:58.528917 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.528891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:49:58.529052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.528994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq" (OuterVolumeSpecName: "kube-api-access-pt2zq") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "kube-api-access-pt2zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:58.529052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token" (OuterVolumeSpecName: "istio-token") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:58.529223 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02dd2157-7b62-478e-b39f-e450ed21b383" (UID: "02dd2157-7b62-478e-b39f-e450ed21b383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:49:58.529879 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529852    2577 generic.go:356] "Generic (PLEG): container finished" podID="02dd2157-7b62-478e-b39f-e450ed21b383" containerID="eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a" exitCode=0
Sep 10 13:49:58.529991 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529898    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"
Sep 10 13:49:58.529991 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerDied","Data":"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"}
Sep 10 13:49:58.529991 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b" event={"ID":"02dd2157-7b62-478e-b39f-e450ed21b383","Type":"ContainerDied","Data":"790a8b3b05faf66c64ea2f9157f1238fdbb9d68001db5b102ba193e975ce4fb0"}
Sep 10 13:49:58.530150 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.529999    2577 scope.go:117] "RemoveContainer" containerID="eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"
Sep 10 13:49:58.550007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.549988    2577 scope.go:117] "RemoveContainer" containerID="e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"
Sep 10 13:49:58.558586 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.558565    2577 scope.go:117] "RemoveContainer" containerID="eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"
Sep 10 13:49:58.558873 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:49:58.558851    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a\": container with ID starting with eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a not found: ID does not exist" containerID="eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"
Sep 10 13:49:58.558945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.558881    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a"} err="failed to get container status \"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a\": rpc error: code = NotFound desc = could not find container \"eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a\": container with ID starting with eca3d34678a5171d3baaff93c75ce272bf393b375ebd23cca91532051cc8e58a not found: ID does not exist"
Sep 10 13:49:58.558945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.558899    2577 scope.go:117] "RemoveContainer" containerID="e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"
Sep 10 13:49:58.559201 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:49:58.559178    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf\": container with ID starting with e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf not found: ID does not exist" containerID="e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"
Sep 10 13:49:58.559251 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.559206    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf"} err="failed to get container status \"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf\": rpc error: code = NotFound desc = could not find container \"e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf\": container with ID starting with e02536a90bc82c930c2160687c87e88b505ac4f73de436dfdbd09a518a132fdf not found: ID does not exist"
Sep 10 13:49:58.571805 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.571779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:49:58.597258 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.597230    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5d0c07-openshift-default-66b4f48476-58g2b"]
Sep 10 13:49:58.628272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628241    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628269    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628279    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628287    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628297    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2zq\" (UniqueName: \"kubernetes.io/projected/02dd2157-7b62-478e-b39f-e450ed21b383-kube-api-access-pt2zq\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628308    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02dd2157-7b62-478e-b39f-e450ed21b383-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:49:58.628521 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:49:58.628318    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02dd2157-7b62-478e-b39f-e450ed21b383-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:00.063953 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:00.063920    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" path="/var/lib/kubelet/pods/02dd2157-7b62-478e-b39f-e450ed21b383/volumes"
Sep 10 13:50:08.654269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:08.654230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:50:10.873609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.873565    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:50:10.874126 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" containerName="maas-api"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874129    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" containerName="maas-api"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874152    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874161    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874171    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874179    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874200    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" containerName="main"
Sep 10 13:50:10.874211 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" containerName="main"
Sep 10 13:50:10.874580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874307    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874319    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02dd2157-7b62-478e-b39f-e450ed21b383" containerName="istio-proxy"
Sep 10 13:50:10.874580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874330    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82aecc3d-12e3-45c3-bf51-6e04581b82f8" containerName="main"
Sep 10 13:50:10.874580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.874340    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e254439-dcbf-43e1-ab72-e3a6e29e2355" containerName="maas-api"
Sep 10 13:50:10.877724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.877702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:10.880127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.880109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3fe902b6-openshift-default-dockercfg-67rrp\""
Sep 10 13:50:10.896583 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:10.896556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:50:11.049860 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.049817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.049874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.049906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.049927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.049956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.050022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050215 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.050059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050215 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.050135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.050215 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.050156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9g7\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.151783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9g7\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.151996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152053 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.152015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.152093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152533 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.152487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.152782 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.152762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.153834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.153802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.154310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.154292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.162755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.162728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9g7\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.162894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.162736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token\") pod \"e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") " pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.187656 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.187629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:11.506835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.506807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:50:11.509351 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:50:11.509324    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff6b9bf_ccdf_4e0f_ba80_1539ceadfe3d.slice/crio-2a6a66d50970157c0a1743cc540eb590b95230f9d654a3803b6b12fc13218917 WatchSource:0}: Error finding container 2a6a66d50970157c0a1743cc540eb590b95230f9d654a3803b6b12fc13218917: Status 404 returned error can't find the container with id 2a6a66d50970157c0a1743cc540eb590b95230f9d654a3803b6b12fc13218917
Sep 10 13:50:11.511708 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.511667    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:11.511798 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.511757    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:11.511839 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.511801    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:11.598732 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.598692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerStarted","Data":"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"}
Sep 10 13:50:11.598890 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.598763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerStarted","Data":"2a6a66d50970157c0a1743cc540eb590b95230f9d654a3803b6b12fc13218917"}
Sep 10 13:50:11.638430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:11.638364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podStartSLOduration=1.638341292 podStartE2EDuration="1.638341292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:11.629344345 +0000 UTC m=+1296.084219508" watchObservedRunningTime="2026-09-10 13:50:11.638341292 +0000 UTC m=+1296.093216455"
Sep 10 13:50:12.188766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:12.188729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:12.193766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:12.193737    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:12.604057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:12.603976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:12.605042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:12.605022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:15.367821 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.367787    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:50:15.371414 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.371388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.374423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.374402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3fe902b6-serving-cert\""
Sep 10 13:50:15.387316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.387287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.387471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.387396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.387471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.387434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.387619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.387474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kzn\" (UniqueName: \"kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.390342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.390317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:50:15.488096 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.488054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.488299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.488163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.488299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.488193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.488299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.488239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kzn\" (UniqueName: \"kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.488436 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:50:15.488348    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-serving-cert: secret "maas-api-e2e-mt-3fe902b6-serving-cert" not found
Sep 10 13:50:15.488436 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:50:15.488432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls podName:d6dbdac6-fa63-460e-905b-75bc43fc9a96 nodeName:}" failed. No retries permitted until 2026-09-10 13:50:15.988410502 +0000 UTC m=+1300.443285680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls") pod "maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" (UID: "d6dbdac6-fa63-460e-905b-75bc43fc9a96") : secret "maas-api-e2e-mt-3fe902b6-serving-cert" not found
Sep 10 13:50:15.488961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.488938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.491075 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.491044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.502741 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.502710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kzn\" (UniqueName: \"kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.990615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.990575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:15.993052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:15.993024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") pod \"maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:16.282291 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:16.282188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:16.574987 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:16.574962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:50:16.577381 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:50:16.577344    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dbdac6_fa63_460e_905b_75bc43fc9a96.slice/crio-764afaf848dd7fc6a1df277594c02264ef72373dba932ffad28d30fe7e88217b WatchSource:0}: Error finding container 764afaf848dd7fc6a1df277594c02264ef72373dba932ffad28d30fe7e88217b: Status 404 returned error can't find the container with id 764afaf848dd7fc6a1df277594c02264ef72373dba932ffad28d30fe7e88217b
Sep 10 13:50:16.627954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:16.627913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" event={"ID":"d6dbdac6-fa63-460e-905b-75bc43fc9a96","Type":"ContainerStarted","Data":"764afaf848dd7fc6a1df277594c02264ef72373dba932ffad28d30fe7e88217b"}
Sep 10 13:50:17.634657 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:17.634616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" event={"ID":"d6dbdac6-fa63-460e-905b-75bc43fc9a96","Type":"ContainerStarted","Data":"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"}
Sep 10 13:50:17.635095 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:17.634693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:17.659558 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:17.659485    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" podStartSLOduration=1.7729059550000001 podStartE2EDuration="2.659468437s" totalImagesPullingTime="886.562482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:17.657788484 +0000 UTC m=+1302.112663648" watchObservedRunningTime="2026-09-10 13:50:17.659468437 +0000 UTC m=+1302.114343638"
Sep 10 13:50:23.063682 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.063643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:50:23.645470 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.645438    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:23.737383 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.737348    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:23.741542 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.741500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.744965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.744801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9aacfe\"/\"kube-root-ca.crt\""
Sep 10 13:50:23.744965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.744818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9aacfe\"/\"default-dockercfg-q9h9b\""
Sep 10 13:50:23.744965 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.744924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-9aacfe\"/\"openshift-service-ca.crt\""
Sep 10 13:50:23.745180 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.744983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-9aacfe\"/\"noten-model-9aacfe-kserve-self-signed-certs\""
Sep 10 13:50:23.759105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.759072    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:23.768969 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.768939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.769101 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.768980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.769101 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.769064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.769101 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.769095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.769232 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.769124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwct\" (UniqueName: \"kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.769329 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.769309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.870755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwct\" (UniqueName: \"kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.870945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.870945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.870945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.871122 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.871122 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.870984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.871269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.871239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.871346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.871250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.871403 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.871349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.873284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.873265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.873381 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.873363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:23.880453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:23.880434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwct\" (UniqueName: \"kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct\") pod \"noten-model-9aacfe-kserve-5b57b479d7-sxqn7\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") " pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:24.051901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:24.051811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:24.347072 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:24.347038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:24.349294 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:50:24.349264    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc39a808b_0869_4aa5_9d0d_8f24ddcba491.slice/crio-2c044908713885adf32ca03c815c75eb67964c84142781b410825fe9d4659470 WatchSource:0}: Error finding container 2c044908713885adf32ca03c815c75eb67964c84142781b410825fe9d4659470: Status 404 returned error can't find the container with id 2c044908713885adf32ca03c815c75eb67964c84142781b410825fe9d4659470
Sep 10 13:50:24.670495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:24.670449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" event={"ID":"c39a808b-0869-4aa5-9d0d-8f24ddcba491","Type":"ContainerStarted","Data":"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"}
Sep 10 13:50:24.670495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:24.670498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" event={"ID":"c39a808b-0869-4aa5-9d0d-8f24ddcba491","Type":"ContainerStarted","Data":"2c044908713885adf32ca03c815c75eb67964c84142781b410825fe9d4659470"}
Sep 10 13:50:24.692079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:24.692027    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" podStartSLOduration=1.692008182 podStartE2EDuration="1.692008182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:50:24.689426042 +0000 UTC m=+1309.144301215" watchObservedRunningTime="2026-09-10 13:50:24.692008182 +0000 UTC m=+1309.146883346"
Sep 10 13:50:30.453324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:30.452984    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:31.523021 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:31.522985    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:50:33.721834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.721754    2577 generic.go:356] "Generic (PLEG): container finished" podID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerID="af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116" exitCode=0
Sep 10 13:50:33.722231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.721827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerDied","Data":"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"}
Sep 10 13:50:33.722297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.722282    2577 scope.go:117] "RemoveContainer" containerID="af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"
Sep 10 13:50:33.723388 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.723345    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:33.723487 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.723433    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:33.723487 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:33.723468    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:50:34.052986 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:34.052948    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:34.728502 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:34.728459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerStarted","Data":"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"}
Sep 10 13:50:34.751311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:34.751256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:50:35.188387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:35.188344    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:35.188707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:35.188683    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 13:50:35.188768 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:35.188732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 13:50:36.188821 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:36.188782    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 13:50:36.189244 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:36.188852    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 13:50:37.188633 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:37.188591    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 13:50:37.188830 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:37.188675    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 13:50:38.188411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:38.188370    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 13:50:38.188865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:38.188447    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 13:50:39.189060 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:39.189024    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Sep 10 13:50:39.189475 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:39.189088    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Sep 10 13:50:40.268306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:40.268218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:40.268802 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:40.268736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:40.269554 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:40.269536    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:50:42.587856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.587823    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:50:42.588316 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.588048    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" podUID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" containerName="maas-api" containerID="cri-o://a56d46353d459f0aa5543c793c35465c4bb190d20f68f54721dc5dca456ff3ec" gracePeriod=30
Sep 10 13:50:42.787589 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.787556    2577 generic.go:356] "Generic (PLEG): container finished" podID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" containerID="a56d46353d459f0aa5543c793c35465c4bb190d20f68f54721dc5dca456ff3ec" exitCode=0
Sep 10 13:50:42.787784 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.787635    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" event={"ID":"9759f70e-dacb-45d7-be59-ca0a33dc4df3","Type":"ContainerDied","Data":"a56d46353d459f0aa5543c793c35465c4bb190d20f68f54721dc5dca456ff3ec"}
Sep 10 13:50:42.837788 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.837762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:50:42.975706 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.975670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs\") pod \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") "
Sep 10 13:50:42.975899 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.975728    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") pod \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") "
Sep 10 13:50:42.975899 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.975786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca\") pod \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") "
Sep 10 13:50:42.975899 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.975810    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng65d\" (UniqueName: \"kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d\") pod \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\" (UID: \"9759f70e-dacb-45d7-be59-ca0a33dc4df3\") "
Sep 10 13:50:42.976218 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.976189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9759f70e-dacb-45d7-be59-ca0a33dc4df3" (UID: "9759f70e-dacb-45d7-be59-ca0a33dc4df3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:50:42.978001 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.977966    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9759f70e-dacb-45d7-be59-ca0a33dc4df3" (UID: "9759f70e-dacb-45d7-be59-ca0a33dc4df3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:42.978111 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.978031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9759f70e-dacb-45d7-be59-ca0a33dc4df3" (UID: "9759f70e-dacb-45d7-be59-ca0a33dc4df3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:42.978324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:42.978302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d" (OuterVolumeSpecName: "kube-api-access-ng65d") pod "9759f70e-dacb-45d7-be59-ca0a33dc4df3" (UID: "9759f70e-dacb-45d7-be59-ca0a33dc4df3"). InnerVolumeSpecName "kube-api-access-ng65d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:43.077471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.077424    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:43.077471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.077466    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9759f70e-dacb-45d7-be59-ca0a33dc4df3-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:43.077471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.077477    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng65d\" (UniqueName: \"kubernetes.io/projected/9759f70e-dacb-45d7-be59-ca0a33dc4df3-kube-api-access-ng65d\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:43.077471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.077485    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9759f70e-dacb-45d7-be59-ca0a33dc4df3-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:43.794209 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.794180    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"
Sep 10 13:50:43.794209 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.794199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622" event={"ID":"9759f70e-dacb-45d7-be59-ca0a33dc4df3","Type":"ContainerDied","Data":"ed7ff474d1b2696e6b93f1f6b7f4c601355eadb190dfcfdb18df36b4061f0432"}
Sep 10 13:50:43.794714 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.794255    2577 scope.go:117] "RemoveContainer" containerID="a56d46353d459f0aa5543c793c35465c4bb190d20f68f54721dc5dca456ff3ec"
Sep 10 13:50:43.817363 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.817334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:50:43.822287 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:43.822259    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-86fdcf-8564c45858-kl622"]
Sep 10 13:50:44.064756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:44.064672    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" path="/var/lib/kubelet/pods/9759f70e-dacb-45d7-be59-ca0a33dc4df3/volumes"
Sep 10 13:50:45.474900 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.474853    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" podUID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" containerName="main" containerID="cri-o://553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e" gracePeriod=45
Sep 10 13:50:45.722411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.722382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:45.806423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.806328    2577 generic.go:356] "Generic (PLEG): container finished" podID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" containerID="553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e" exitCode=0
Sep 10 13:50:45.806423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.806374    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"
Sep 10 13:50:45.806423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.806408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" event={"ID":"c39a808b-0869-4aa5-9d0d-8f24ddcba491","Type":"ContainerDied","Data":"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"}
Sep 10 13:50:45.806733 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.806452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7" event={"ID":"c39a808b-0869-4aa5-9d0d-8f24ddcba491","Type":"ContainerDied","Data":"2c044908713885adf32ca03c815c75eb67964c84142781b410825fe9d4659470"}
Sep 10 13:50:45.806733 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.806471    2577 scope.go:117] "RemoveContainer" containerID="553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"
Sep 10 13:50:45.814722 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.814701    2577 scope.go:117] "RemoveContainer" containerID="553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"
Sep 10 13:50:45.815035 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:50:45.815003    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e\": container with ID starting with 553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e not found: ID does not exist" containerID="553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"
Sep 10 13:50:45.815138 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.815040    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e"} err="failed to get container status \"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e\": rpc error: code = NotFound desc = could not find container \"553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e\": container with ID starting with 553a8e293fb3d65b437652015f24bdf36df5a8409a7a78e789c26448d4442e0e not found: ID does not exist"
Sep 10 13:50:45.874168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.874131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:50:45.906299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcwct\" (UniqueName: \"kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906495 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906698 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906688    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache" (OuterVolumeSpecName: "model-cache") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:45.906763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs\") pod \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\" (UID: \"c39a808b-0869-4aa5-9d0d-8f24ddcba491\") "
Sep 10 13:50:45.906903 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.906881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:45.907133 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.907111    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:45.907246 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.907135    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:45.907342 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.907323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home" (OuterVolumeSpecName: "home") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:45.909500 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.909463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm" (OuterVolumeSpecName: "dshm") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:45.909872 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.909837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct" (OuterVolumeSpecName: "kube-api-access-hcwct") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "kube-api-access-hcwct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:45.910284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:45.910236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c39a808b-0869-4aa5-9d0d-8f24ddcba491" (UID: "c39a808b-0869-4aa5-9d0d-8f24ddcba491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:46.008299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.008254    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.008299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.008289    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c39a808b-0869-4aa5-9d0d-8f24ddcba491-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.008299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.008302    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c39a808b-0869-4aa5-9d0d-8f24ddcba491-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.008601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.008314    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwct\" (UniqueName: \"kubernetes.io/projected/c39a808b-0869-4aa5-9d0d-8f24ddcba491-kube-api-access-hcwct\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.130412 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.130373    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:46.136097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.136068    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-9aacfe/noten-model-9aacfe-kserve-5b57b479d7-sxqn7"]
Sep 10 13:50:46.545955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.545916    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" podUID="7de09390-66d4-4174-b6e1-f507c345b174" containerName="main" containerID="cri-o://1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde" gracePeriod=45
Sep 10 13:50:46.800773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.800707    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:50:46.812318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.812285    2577 generic.go:356] "Generic (PLEG): container finished" podID="7de09390-66d4-4174-b6e1-f507c345b174" containerID="1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde" exitCode=0
Sep 10 13:50:46.812482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.812334    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"
Sep 10 13:50:46.812482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.812377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" event={"ID":"7de09390-66d4-4174-b6e1-f507c345b174","Type":"ContainerDied","Data":"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"}
Sep 10 13:50:46.812482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.812417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8" event={"ID":"7de09390-66d4-4174-b6e1-f507c345b174","Type":"ContainerDied","Data":"e020d94fcf3cbd2c2cb635de06e821964a9a136c776787c8b0fc39a038e72f96"}
Sep 10 13:50:46.812482 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.812436    2577 scope.go:117] "RemoveContainer" containerID="1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"
Sep 10 13:50:46.822417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.822052    2577 scope.go:117] "RemoveContainer" containerID="1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"
Sep 10 13:50:46.823177 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:50:46.823139    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde\": container with ID starting with 1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde not found: ID does not exist" containerID="1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"
Sep 10 13:50:46.823302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.823180    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde"} err="failed to get container status \"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde\": rpc error: code = NotFound desc = could not find container \"1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde\": container with ID starting with 1d2c7bc47779418770aa790dbe0596803e6383ea3204389066f916b39f0a0dde not found: ID does not exist"
Sep 10 13:50:46.919448 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919661 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919647    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lzwj\" (UniqueName: \"kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs\") pod \"7de09390-66d4-4174-b6e1-f507c345b174\" (UID: \"7de09390-66d4-4174-b6e1-f507c345b174\") "
Sep 10 13:50:46.919943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home" (OuterVolumeSpecName: "home") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:46.919943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:46.919943 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.919850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache" (OuterVolumeSpecName: "model-cache") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:46.920098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.920024    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.920098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.920039    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.920098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.920047    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:46.921931 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.921907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm" (OuterVolumeSpecName: "dshm") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:50:46.921931 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.921925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:46.922087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:46.921940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj" (OuterVolumeSpecName: "kube-api-access-8lzwj") pod "7de09390-66d4-4174-b6e1-f507c345b174" (UID: "7de09390-66d4-4174-b6e1-f507c345b174"). InnerVolumeSpecName "kube-api-access-8lzwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:47.020799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:47.020764    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de09390-66d4-4174-b6e1-f507c345b174-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:47.020799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:47.020796    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lzwj\" (UniqueName: \"kubernetes.io/projected/7de09390-66d4-4174-b6e1-f507c345b174-kube-api-access-8lzwj\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:47.020799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:47.020808    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de09390-66d4-4174-b6e1-f507c345b174-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:47.159332 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:47.159293    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:50:47.164476 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:47.164445    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-02fc2e/test-model-02fc2e-kserve-85df95fb5-7vnd8"]
Sep 10 13:50:48.063767 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:48.063722    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de09390-66d4-4174-b6e1-f507c345b174" path="/var/lib/kubelet/pods/7de09390-66d4-4174-b6e1-f507c345b174/volumes"
Sep 10 13:50:48.064298 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:48.064275    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" path="/var/lib/kubelet/pods/c39a808b-0869-4aa5-9d0d-8f24ddcba491/volumes"
Sep 10 13:50:58.317888 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.317851    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:50:58.318372 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.318125    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" podUID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" containerName="maas-api" containerID="cri-o://7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36" gracePeriod=30
Sep 10 13:50:58.584912 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.584884    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:58.626011 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.625979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca\") pod \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") "
Sep 10 13:50:58.626173 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.626018    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2kzn\" (UniqueName: \"kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn\") pod \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") "
Sep 10 13:50:58.626173 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.626077    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") pod \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") "
Sep 10 13:50:58.626173 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.626115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs\") pod \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\" (UID: \"d6dbdac6-fa63-460e-905b-75bc43fc9a96\") "
Sep 10 13:50:58.626468 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.626423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6dbdac6-fa63-460e-905b-75bc43fc9a96" (UID: "d6dbdac6-fa63-460e-905b-75bc43fc9a96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:50:58.628796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.628770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6dbdac6-fa63-460e-905b-75bc43fc9a96" (UID: "d6dbdac6-fa63-460e-905b-75bc43fc9a96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:58.628954 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.628927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6dbdac6-fa63-460e-905b-75bc43fc9a96" (UID: "d6dbdac6-fa63-460e-905b-75bc43fc9a96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:58.629306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.629282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn" (OuterVolumeSpecName: "kube-api-access-x2kzn") pod "d6dbdac6-fa63-460e-905b-75bc43fc9a96" (UID: "d6dbdac6-fa63-460e-905b-75bc43fc9a96"). InnerVolumeSpecName "kube-api-access-x2kzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:58.727243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.727210    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dbdac6-fa63-460e-905b-75bc43fc9a96-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:58.727243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.727240    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kzn\" (UniqueName: \"kubernetes.io/projected/d6dbdac6-fa63-460e-905b-75bc43fc9a96-kube-api-access-x2kzn\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:58.727243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.727249    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:58.727486 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.727258    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dbdac6-fa63-460e-905b-75bc43fc9a96-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:50:58.875731 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.875692    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" containerID="7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36" exitCode=0
Sep 10 13:50:58.875891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.875756    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"
Sep 10 13:50:58.875891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.875781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" event={"ID":"d6dbdac6-fa63-460e-905b-75bc43fc9a96","Type":"ContainerDied","Data":"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"}
Sep 10 13:50:58.875891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.875842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf" event={"ID":"d6dbdac6-fa63-460e-905b-75bc43fc9a96","Type":"ContainerDied","Data":"764afaf848dd7fc6a1df277594c02264ef72373dba932ffad28d30fe7e88217b"}
Sep 10 13:50:58.875891 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.875864    2577 scope.go:117] "RemoveContainer" containerID="7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"
Sep 10 13:50:58.885453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.885433    2577 scope.go:117] "RemoveContainer" containerID="7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"
Sep 10 13:50:58.885750 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:50:58.885730    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36\": container with ID starting with 7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36 not found: ID does not exist" containerID="7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"
Sep 10 13:50:58.885818 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.885758    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36"} err="failed to get container status \"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36\": rpc error: code = NotFound desc = could not find container \"7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36\": container with ID starting with 7e13d5f0f02623f55d13a3f4ba1af49f6ced7d86d8751fd3b682bed9b913dc36 not found: ID does not exist"
Sep 10 13:50:58.912401 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.912368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:50:58.916969 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:50:58.916947    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe902b6-78d5f57f6-hzkdf"]
Sep 10 13:51:00.064048 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:00.064015    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" path="/var/lib/kubelet/pods/d6dbdac6-fa63-460e-905b-75bc43fc9a96/volumes"
Sep 10 13:51:01.577303 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:01.577221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:51:02.793269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:02.793235    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:51:02.793699 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:02.793472    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" podUID="2e828ce8-fa01-459c-add8-19566d3f8217" containerName="maas-api" containerID="cri-o://4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d" gracePeriod=30
Sep 10 13:51:03.039663 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.039628    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:51:03.168107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs\") pod \"2e828ce8-fa01-459c-add8-19566d3f8217\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") "
Sep 10 13:51:03.168107 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca\") pod \"2e828ce8-fa01-459c-add8-19566d3f8217\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") "
Sep 10 13:51:03.168379 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") pod \"2e828ce8-fa01-459c-add8-19566d3f8217\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") "
Sep 10 13:51:03.168379 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6dpr\" (UniqueName: \"kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr\") pod \"2e828ce8-fa01-459c-add8-19566d3f8217\" (UID: \"2e828ce8-fa01-459c-add8-19566d3f8217\") "
Sep 10 13:51:03.168629 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e828ce8-fa01-459c-add8-19566d3f8217" (UID: "2e828ce8-fa01-459c-add8-19566d3f8217"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:51:03.168789 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.168681    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e828ce8-fa01-459c-add8-19566d3f8217-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:03.170420 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.170396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr" (OuterVolumeSpecName: "kube-api-access-r6dpr") pod "2e828ce8-fa01-459c-add8-19566d3f8217" (UID: "2e828ce8-fa01-459c-add8-19566d3f8217"). InnerVolumeSpecName "kube-api-access-r6dpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:51:03.170836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.170810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e828ce8-fa01-459c-add8-19566d3f8217" (UID: "2e828ce8-fa01-459c-add8-19566d3f8217"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:03.170921 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.170831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e828ce8-fa01-459c-add8-19566d3f8217" (UID: "2e828ce8-fa01-459c-add8-19566d3f8217"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:03.269963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.269924    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:03.269963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.269955    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e828ce8-fa01-459c-add8-19566d3f8217-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:03.269963 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.269964    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6dpr\" (UniqueName: \"kubernetes.io/projected/2e828ce8-fa01-459c-add8-19566d3f8217-kube-api-access-r6dpr\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:03.904393 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.904353    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e828ce8-fa01-459c-add8-19566d3f8217" containerID="4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d" exitCode=0
Sep 10 13:51:03.904878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.904399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"
Sep 10 13:51:03.904878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.904438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" event={"ID":"2e828ce8-fa01-459c-add8-19566d3f8217","Type":"ContainerDied","Data":"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"}
Sep 10 13:51:03.904878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.904499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf" event={"ID":"2e828ce8-fa01-459c-add8-19566d3f8217","Type":"ContainerDied","Data":"a95721d20fdf867d334436c9c7293685856cf6201cfd49fc77149c56a0856e53"}
Sep 10 13:51:03.904878 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.904537    2577 scope.go:117] "RemoveContainer" containerID="4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"
Sep 10 13:51:03.914320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.914297    2577 scope.go:117] "RemoveContainer" containerID="4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"
Sep 10 13:51:03.914669 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:03.914647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d\": container with ID starting with 4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d not found: ID does not exist" containerID="4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"
Sep 10 13:51:03.914744 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.914678    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d"} err="failed to get container status \"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d\": rpc error: code = NotFound desc = could not find container \"4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d\": container with ID starting with 4617232da40683bce4dfd508f4da939baef06e23e3da04f2a5c92b561217c31d not found: ID does not exist"
Sep 10 13:51:03.932418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.932389    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:51:03.935724 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:03.935701    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-02fc2e-8b468bdd-vrcdf"]
Sep 10 13:51:04.063193 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:04.063158    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e828ce8-fa01-459c-add8-19566d3f8217" path="/var/lib/kubelet/pods/2e828ce8-fa01-459c-add8-19566d3f8217/volumes"
Sep 10 13:51:07.386643 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:07.386607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:51:07.387104 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:07.386820    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" containerID="cri-o://42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965" gracePeriod=30
Sep 10 13:51:09.343735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:09.343692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:51:10.271284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:10.271240    2577 patch_prober.go:28] interesting pod/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:51:10.271470 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:10.271329    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:51:12.641601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.641570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:51:12.759560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759442    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:51:12.759863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.760188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k9g7\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7\") pod \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\" (UID: \"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d\") "
Sep 10 13:51:12.760188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.759916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:51:12.760188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:51:12.760188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:51:12.760453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760258    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.760453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760278    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.760453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760292    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.760453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760307    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.760453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.760373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:51:12.761857 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.761829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:51:12.761999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.761854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:51:12.761999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.761872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:51:12.761999 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.761907    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7" (OuterVolumeSpecName: "kube-api-access-9k9g7") pod "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" (UID: "dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d"). InnerVolumeSpecName "kube-api-access-9k9g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:51:12.861153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.861113    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.861153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.861147    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.861153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.861155    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.861153 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.861163    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.861422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.861171    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k9g7\" (UniqueName: \"kubernetes.io/projected/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d-kube-api-access-9k9g7\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:51:12.951334 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.951297    2577 generic.go:356] "Generic (PLEG): container finished" podID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerID="42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965" exitCode=0
Sep 10 13:51:12.951547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.951406    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"
Sep 10 13:51:12.951547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.951427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerDied","Data":"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"}
Sep 10 13:51:12.951547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.951487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w" event={"ID":"dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d","Type":"ContainerDied","Data":"2a6a66d50970157c0a1743cc540eb590b95230f9d654a3803b6b12fc13218917"}
Sep 10 13:51:12.951547 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.951537    2577 scope.go:117] "RemoveContainer" containerID="42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"
Sep 10 13:51:12.960775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.960756    2577 scope.go:117] "RemoveContainer" containerID="af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"
Sep 10 13:51:12.968296 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.968277    2577 scope.go:117] "RemoveContainer" containerID="42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"
Sep 10 13:51:12.968584 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:12.968553    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965\": container with ID starting with 42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965 not found: ID does not exist" containerID="42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"
Sep 10 13:51:12.968690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.968591    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965"} err="failed to get container status \"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965\": rpc error: code = NotFound desc = could not find container \"42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965\": container with ID starting with 42a0a44c62fea949d9803f7e0f3b2067c6a96dadeed64ce32579d4dcb381f965 not found: ID does not exist"
Sep 10 13:51:12.968690 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.968608    2577 scope.go:117] "RemoveContainer" containerID="af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"
Sep 10 13:51:12.968866 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:12.968850    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116\": container with ID starting with af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116 not found: ID does not exist" containerID="af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"
Sep 10 13:51:12.968913 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.968868    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116"} err="failed to get container status \"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116\": rpc error: code = NotFound desc = could not find container \"af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116\": container with ID starting with af15c5b6e2be6ab511fdf2b447f6c6da60747278d6378d2b0e2a61fcee6b2116 not found: ID does not exist"
Sep 10 13:51:12.973293 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.973273    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:51:12.977447 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:12.977427    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3fe902b6-openshift-default-5bdcc847cb-6s95w"]
Sep 10 13:51:14.063325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:14.063286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" path="/var/lib/kubelet/pods/dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d/volumes"
Sep 10 13:51:15.984277 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984242    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" containerName="maas-api"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984639    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" containerName="maas-api"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de09390-66d4-4174-b6e1-f507c345b174" containerName="main"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984656    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de09390-66d4-4174-b6e1-f507c345b174" containerName="main"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984670    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984675    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984682    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" containerName="main"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984687    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" containerName="main"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984699    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984704    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984713    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e828ce8-fa01-459c-add8-19566d3f8217" containerName="maas-api"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984719    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e828ce8-fa01-459c-add8-19566d3f8217" containerName="maas-api"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" containerName="maas-api"
Sep 10 13:51:15.984765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984729    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" containerName="maas-api"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de09390-66d4-4174-b6e1-f507c345b174" containerName="main"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984796    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9759f70e-dacb-45d7-be59-ca0a33dc4df3" containerName="maas-api"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984801    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dbdac6-fa63-460e-905b-75bc43fc9a96" containerName="maas-api"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984805    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e828ce8-fa01-459c-add8-19566d3f8217" containerName="maas-api"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984811    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff6b9bf-ccdf-4e0f-ba80-1539ceadfe3d" containerName="istio-proxy"
Sep 10 13:51:15.985241 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.984817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c39a808b-0869-4aa5-9d0d-8f24ddcba491" containerName="main"
Sep 10 13:51:15.987909 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.987894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:15.991594 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:15.991572    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7a11e9-serving-cert\""
Sep 10 13:51:16.014615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.014589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 13:51:16.091367 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.091331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.091565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.091374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.091565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.091392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64lj\" (UniqueName: \"kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.091565 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.091551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.192339 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.192303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.192563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.192358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.192563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.192382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.192563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.192399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64lj\" (UniqueName: \"kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.192563 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:16.192452    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-serving-cert: secret "maas-api-e2e-shared-a-w3-7a11e9-serving-cert" not found
Sep 10 13:51:16.192563 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:16.192563    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls podName:f914eea1-c419-4c92-ba25-edf4a5505a93 nodeName:}" failed. No retries permitted until 2026-09-10 13:51:16.69254166 +0000 UTC m=+1361.147416809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls") pod "maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" (UID: "f914eea1-c419-4c92-ba25-edf4a5505a93") : secret "maas-api-e2e-shared-a-w3-7a11e9-serving-cert" not found
Sep 10 13:51:16.193169 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.193143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.194861 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.194844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.206216 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.206180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64lj\" (UniqueName: \"kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.698054 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.698009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.700541 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.700486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:16.898238 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:16.898196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:17.201169 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:17.201129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 13:51:17.203842 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:51:17.203526    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf914eea1_c419_4c92_ba25_edf4a5505a93.slice/crio-8e4901b13f9d5ae2f940a64efc6ba6ff2773cf713cbfba9dbfd270d891bfac84 WatchSource:0}: Error finding container 8e4901b13f9d5ae2f940a64efc6ba6ff2773cf713cbfba9dbfd270d891bfac84: Status 404 returned error can't find the container with id 8e4901b13f9d5ae2f940a64efc6ba6ff2773cf713cbfba9dbfd270d891bfac84
Sep 10 13:51:17.981159 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:17.981127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" event={"ID":"f914eea1-c419-4c92-ba25-edf4a5505a93","Type":"ContainerStarted","Data":"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"}
Sep 10 13:51:17.981324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:17.981175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" event={"ID":"f914eea1-c419-4c92-ba25-edf4a5505a93","Type":"ContainerStarted","Data":"8e4901b13f9d5ae2f940a64efc6ba6ff2773cf713cbfba9dbfd270d891bfac84"}
Sep 10 13:51:17.981324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:17.981207    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:18.002783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:18.002724    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" podStartSLOduration=2.350076095 podStartE2EDuration="3.002708616s" totalImagesPullingTime="652.632521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:18.000578592 +0000 UTC m=+1362.455453759" watchObservedRunningTime="2026-09-10 13:51:18.002708616 +0000 UTC m=+1362.457583778"
Sep 10 13:51:20.249421 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:20.249384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:51:23.992315 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:23.992284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 13:51:24.323721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.323641    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:51:24.327243 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.327226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.329494 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.329465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-83c318-openshift-default-dockercfg-8kg7s\""
Sep 10 13:51:24.341065 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.341036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:51:24.466886 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.466845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.466886 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.466888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.466929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.466948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k999p\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.466976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467137 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.467088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.467195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.467232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.467346 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.467268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568268 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k999p\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.568781 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.569000 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.568979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.569096 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.569018    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.569096 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.569067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.570569 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.570545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.570869 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.570851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.576619 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.576559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.576705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.576688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k999p\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p\") pod \"e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") " pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.637764 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.637722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:24.917020 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.916936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:51:24.919285 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:51:24.919241    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9f321f_c6be_4cd9_9454_b20c4436ce9f.slice/crio-ec32cf3c02d78ef1d3c15e07e2db5c138a666d72c9257eeec4d94a528109cbf2 WatchSource:0}: Error finding container ec32cf3c02d78ef1d3c15e07e2db5c138a666d72c9257eeec4d94a528109cbf2: Status 404 returned error can't find the container with id ec32cf3c02d78ef1d3c15e07e2db5c138a666d72c9257eeec4d94a528109cbf2
Sep 10 13:51:24.921564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.921520    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:24.921640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.921624    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:24.921689 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:24.921672    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:25.019116 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:25.019074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerStarted","Data":"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"}
Sep 10 13:51:25.019554 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:25.019155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerStarted","Data":"ec32cf3c02d78ef1d3c15e07e2db5c138a666d72c9257eeec4d94a528109cbf2"}
Sep 10 13:51:25.046819 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:25.046753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podStartSLOduration=1.046728459 podStartE2EDuration="1.046728459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:25.045619444 +0000 UTC m=+1369.500494608" watchObservedRunningTime="2026-09-10 13:51:25.046728459 +0000 UTC m=+1369.501603622"
Sep 10 13:51:25.638045 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:25.638005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:25.643544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:25.643502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:26.024201 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:26.024109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:26.025295 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:26.025277    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:28.240159 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.240119    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:51:28.243939 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.243914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.247007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.246981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-83c318-serving-cert\""
Sep 10 13:51:28.257279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.257244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:51:28.409165 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.409119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.409430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.409181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.409430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.409204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqmd\" (UniqueName: \"kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.409430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.409286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.510946 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.510840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.511136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.510955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.511136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.511006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.511136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.511031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqmd\" (UniqueName: \"kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.511299 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:28.511142    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-serving-cert: secret "maas-api-e2e-shared-a-w2-83c318-serving-cert" not found
Sep 10 13:51:28.511299 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:28.511228    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls podName:3ac798a4-d360-4bd1-84c8-591a37e941e8 nodeName:}" failed. No retries permitted until 2026-09-10 13:51:29.011204989 +0000 UTC m=+1373.466080144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls") pod "maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" (UID: "3ac798a4-d360-4bd1-84c8-591a37e941e8") : secret "maas-api-e2e-shared-a-w2-83c318-serving-cert" not found
Sep 10 13:51:28.511745 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.511724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.513272 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.513254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:28.520076 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:28.520050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqmd\" (UniqueName: \"kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:29.016901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:29.016861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:29.019540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:29.019485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:29.157019 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:29.156976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:29.440380 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:29.440342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:51:29.442143 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:51:29.442115    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac798a4_d360_4bd1_84c8_591a37e941e8.slice/crio-45d792a6b7886dc02f0a82f9792219e4befa3eb003d9c29301d3a249726f743a WatchSource:0}: Error finding container 45d792a6b7886dc02f0a82f9792219e4befa3eb003d9c29301d3a249726f743a: Status 404 returned error can't find the container with id 45d792a6b7886dc02f0a82f9792219e4befa3eb003d9c29301d3a249726f743a
Sep 10 13:51:30.045668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:30.045632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" event={"ID":"3ac798a4-d360-4bd1-84c8-591a37e941e8","Type":"ContainerStarted","Data":"45d792a6b7886dc02f0a82f9792219e4befa3eb003d9c29301d3a249726f743a"}
Sep 10 13:51:31.051432 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:31.051394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" event={"ID":"3ac798a4-d360-4bd1-84c8-591a37e941e8","Type":"ContainerStarted","Data":"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"}
Sep 10 13:51:31.051889 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:31.051484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:31.073373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:31.073320    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" podStartSLOduration=2.2011804919999998 podStartE2EDuration="3.073305299s" totalImagesPullingTime="872.124807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:31.071094027 +0000 UTC m=+1375.525969192" watchObservedRunningTime="2026-09-10 13:51:31.073305299 +0000 UTC m=+1375.528180462"
Sep 10 13:51:31.844923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:31.844883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:51:37.068306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:37.068267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:51:41.147853 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:41.147810    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:45544->10.134.0.56:15021: read: connection reset by peer" start-of-body=
Sep 10 13:51:41.148240 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:41.147878    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:45544->10.134.0.56:15021: read: connection reset by peer"
Sep 10 13:51:42.118852 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.118818    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerID="5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7" exitCode=0
Sep 10 13:51:42.119038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.118890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerDied","Data":"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"}
Sep 10 13:51:42.119325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.119312    2577 scope.go:117] "RemoveContainer" containerID="5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"
Sep 10 13:51:42.120455 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.120416    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:42.120580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.120498    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:42.120580 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:42.120549    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:51:43.126401 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:43.126366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerStarted","Data":"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"}
Sep 10 13:51:43.638294 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:43.638250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:43.638594 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:43.638563    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:51:43.638692 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:43.638622    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:51:44.638727 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:44.638683    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:51:44.639373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:44.639333    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:51:45.639306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:45.639259    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:51:45.639956 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:45.639914    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:51:46.638836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:46.638787    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:51:46.639163 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:46.639114    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:51:47.639004 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:47.638960    2577 patch_prober.go:28] interesting pod/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 13:51:47.639548 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:47.639035    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 13:51:48.642809 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:48.642768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:48.643180 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:48.643132    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:48.643811 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:48.643790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:51:51.016776 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.016741    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 13:51:51.019242 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.019225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.021919 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.021895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-a00003-serving-cert\""
Sep 10 13:51:51.032908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.032886    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 13:51:51.135152 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.135121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.135376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.135180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.135376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.135278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.135376 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.135316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv74m\" (UniqueName: \"kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.236063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.236023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.236269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.236090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.236269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.236129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.236269 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.236150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv74m\" (UniqueName: \"kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.236269 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:51.236260    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-serving-cert: secret "maas-api-e2e-shared-b-w3-a00003-serving-cert" not found
Sep 10 13:51:51.236484 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:51.236352    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls podName:fe88808a-4d8e-44b8-b4c1-0e6d13982b7a nodeName:}" failed. No retries permitted until 2026-09-10 13:51:51.736329667 +0000 UTC m=+1396.191204822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" (UID: "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a") : secret "maas-api-e2e-shared-b-w3-a00003-serving-cert" not found
Sep 10 13:51:51.236751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.236731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.238544 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.238523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.246769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.246744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv74m\" (UniqueName: \"kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.741749 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.741714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.744084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.744054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:51.929306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:51.929264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:52.202632 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:52.202604    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 13:51:52.204388 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:51:52.204362    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe88808a_4d8e_44b8_b4c1_0e6d13982b7a.slice/crio-983a8a179781f2e118ffdcbb88f48498edcb85303c0364e907e619aafc37d03f WatchSource:0}: Error finding container 983a8a179781f2e118ffdcbb88f48498edcb85303c0364e907e619aafc37d03f: Status 404 returned error can't find the container with id 983a8a179781f2e118ffdcbb88f48498edcb85303c0364e907e619aafc37d03f
Sep 10 13:51:53.190988 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:53.190953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" event={"ID":"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a","Type":"ContainerStarted","Data":"b19dd7c22bbe2462e7dc03d65dbdc8b11c8b807fe06e5e78433cb0521f75301a"}
Sep 10 13:51:53.191174 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:53.191013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" event={"ID":"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a","Type":"ContainerStarted","Data":"983a8a179781f2e118ffdcbb88f48498edcb85303c0364e907e619aafc37d03f"}
Sep 10 13:51:53.191174 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:53.191051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:53.211837 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:53.211782    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" podStartSLOduration=2.437844985 podStartE2EDuration="3.211765664s" totalImagesPullingTime="773.920679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:53.209800836 +0000 UTC m=+1397.664675999" watchObservedRunningTime="2026-09-10 13:51:53.211765664 +0000 UTC m=+1397.666640826"
Sep 10 13:51:56.457769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:56.457728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:51:57.237575 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.237535    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:51:57.243312 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.243289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.245502 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.245482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a310dd-serving-cert\""
Sep 10 13:51:57.253393 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.253367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:51:57.399893 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.399850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.400097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.399899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.400097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.399994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vfv\" (UniqueName: \"kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.400097 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.400078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.501041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.500945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vfv\" (UniqueName: \"kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.501562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.501039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.501562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.501085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.501562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.501112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.501562 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:57.501210    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-serving-cert: secret "maas-api-e2e-shared-b-w2-a310dd-serving-cert" not found
Sep 10 13:51:57.501562 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:51:57.501302    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls podName:22e8d7c8-6241-4f33-9cc6-feed7996278b nodeName:}" failed. No retries permitted until 2026-09-10 13:51:58.001280426 +0000 UTC m=+1402.456155573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" (UID: "22e8d7c8-6241-4f33-9cc6-feed7996278b") : secret "maas-api-e2e-shared-b-w2-a310dd-serving-cert" not found
Sep 10 13:51:57.501823 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.501804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.503408 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.503391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:57.520084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:57.520056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vfv\" (UniqueName: \"kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:58.005157 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:58.005118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:58.007545 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:58.007496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:58.156950 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:58.156912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:58.457845 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:58.457817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:51:58.458973 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:51:58.458943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e8d7c8_6241_4f33_9cc6_feed7996278b.slice/crio-1deb017767649d5bcf65aacbaf73563f2680ebf007188bb14573736bbf5ed584 WatchSource:0}: Error finding container 1deb017767649d5bcf65aacbaf73563f2680ebf007188bb14573736bbf5ed584: Status 404 returned error can't find the container with id 1deb017767649d5bcf65aacbaf73563f2680ebf007188bb14573736bbf5ed584
Sep 10 13:51:59.201292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:59.201260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 13:51:59.222354 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:59.222313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" event={"ID":"22e8d7c8-6241-4f33-9cc6-feed7996278b","Type":"ContainerStarted","Data":"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"}
Sep 10 13:51:59.222563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:59.222363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" event={"ID":"22e8d7c8-6241-4f33-9cc6-feed7996278b","Type":"ContainerStarted","Data":"1deb017767649d5bcf65aacbaf73563f2680ebf007188bb14573736bbf5ed584"}
Sep 10 13:51:59.222563 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:59.222456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:51:59.243712 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:51:59.243639    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" podStartSLOduration=1.906798107 podStartE2EDuration="2.243615985s" totalImagesPullingTime="336.817878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:51:59.241688804 +0000 UTC m=+1403.696563966" watchObservedRunningTime="2026-09-10 13:51:59.243615985 +0000 UTC m=+1403.698491148"
Sep 10 13:52:05.232561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:05.232498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:52:08.240969 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:08.240935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:52:24.700526 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.700470    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:52:24.710460 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.710429    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.713829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.713801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"kube-root-ca.crt\""
Sep 10 13:52:24.714003 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.713872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"auth-test-model-7a11e9-kserve-self-signed-certs\""
Sep 10 13:52:24.714091 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.713998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"default-dockercfg-2jqwd\""
Sep 10 13:52:24.714228 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.713888    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"openshift-service-ca.crt\""
Sep 10 13:52:24.716167 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.716145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:52:24.766082 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qpw\" (UniqueName: \"kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.766230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.766230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.766230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.766230 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.766366 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.766266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867079 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qpw\" (UniqueName: \"kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867254 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867678 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.867678 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.867644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.869411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.869390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.869687 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.869667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:24.879740 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:24.879710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qpw\" (UniqueName: \"kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw\") pod \"auth-test-model-7a11e9-kserve-678747fd9f-m64qw\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:25.021503 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:25.021416    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:25.308924 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:52:25.308890    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb022141_cfe7_49eb_91a6_c03df264e983.slice/crio-a5f480d1a10e5b0337c662b7194a005352e42ddadf8ef543a66cfad64228de6d WatchSource:0}: Error finding container a5f480d1a10e5b0337c662b7194a005352e42ddadf8ef543a66cfad64228de6d: Status 404 returned error can't find the container with id a5f480d1a10e5b0337c662b7194a005352e42ddadf8ef543a66cfad64228de6d
Sep 10 13:52:25.309472 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:25.309447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:52:25.354936 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:25.354910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" event={"ID":"fb022141-cfe7-49eb-91a6-c03df264e983","Type":"ContainerStarted","Data":"a5f480d1a10e5b0337c662b7194a005352e42ddadf8ef543a66cfad64228de6d"}
Sep 10 13:52:26.361756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:26.361715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" event={"ID":"fb022141-cfe7-49eb-91a6-c03df264e983","Type":"ContainerStarted","Data":"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"}
Sep 10 13:52:26.389382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:26.389330    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" podStartSLOduration=2.389314308 podStartE2EDuration="2.389314308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:26.389010549 +0000 UTC m=+1430.843885886" watchObservedRunningTime="2026-09-10 13:52:26.389314308 +0000 UTC m=+1430.844189467"
Sep 10 13:52:28.631306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:28.631226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:52:35.022347 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:35.022311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:35.022810 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:35.022359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:35.034919 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:35.034894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:35.420441 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:35.420414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:52:44.041310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:52:44.041273    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:53:24.258835 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.258801    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:53:24.262292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.262274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.265445 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.265421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-713ee90e\"/\"e2e-lifecycle-model-713ee90e-kserve-self-signed-certs\""
Sep 10 13:53:24.265608 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.265467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-713ee90e\"/\"kube-root-ca.crt\""
Sep 10 13:53:24.265608 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.265599    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-713ee90e\"/\"openshift-service-ca.crt\""
Sep 10 13:53:24.265735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.265630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-713ee90e\"/\"default-dockercfg-rvdbx\""
Sep 10 13:53:24.272595 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.272572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:53:24.333089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.333284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfzn\" (UniqueName: \"kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.333284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.333284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.333284 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.333423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.333314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434231 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfzn\" (UniqueName: \"kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.434826 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.434809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.436596 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.436569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.436842 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.436822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.445676 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.445642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfzn\" (UniqueName: \"kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn\") pod \"e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") " pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.572555 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.572447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:24.864442 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:24.864406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:53:24.868696 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:53:24.868666    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfcf17e4_a111_44e1_81f9_f5d90e70878c.slice/crio-45f68ec1d868586608e95e996ff30b8d77feecc4fc4b3fe3d15504147758ea82 WatchSource:0}: Error finding container 45f68ec1d868586608e95e996ff30b8d77feecc4fc4b3fe3d15504147758ea82: Status 404 returned error can't find the container with id 45f68ec1d868586608e95e996ff30b8d77feecc4fc4b3fe3d15504147758ea82
Sep 10 13:53:25.653491 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:25.653444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" event={"ID":"cfcf17e4-a111-44e1-81f9-f5d90e70878c","Type":"ContainerStarted","Data":"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"}
Sep 10 13:53:25.653491 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:25.653493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" event={"ID":"cfcf17e4-a111-44e1-81f9-f5d90e70878c","Type":"ContainerStarted","Data":"45f68ec1d868586608e95e996ff30b8d77feecc4fc4b3fe3d15504147758ea82"}
Sep 10 13:53:25.677739 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:25.677684    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" podStartSLOduration=1.6776667669999998 podStartE2EDuration="1.677666767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:53:25.675488999 +0000 UTC m=+1490.130364162" watchObservedRunningTime="2026-09-10 13:53:25.677666767 +0000 UTC m=+1490.132541930"
Sep 10 13:53:29.337038 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:29.337004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:53:34.572613 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:34.572561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:34.573056 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:34.572658    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:34.585219 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:34.585195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:34.711064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:34.711031    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:53:36.106427 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:36.106395    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:53:36.110492 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:36.110472    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:53:44.210322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.210288    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:53:44.213862 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.213837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.216007 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.215979    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"openshift-service-ca.crt\""
Sep 10 13:53:44.216864 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.216847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"kube-root-ca.crt\""
Sep 10 13:53:44.216982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.216865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"default-dockercfg-4cvkz\""
Sep 10 13:53:44.217049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.217030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"auth-test-model-a00003-kserve-self-signed-certs\""
Sep 10 13:53:44.225906 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.225880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:53:44.334083 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.334083 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.334311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95th8\" (UniqueName: \"kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.334311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.334311 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.334410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.334315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435474 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435474 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435761 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95th8\" (UniqueName: \"kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.435901 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.436009 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.435935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.436236 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.436210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.436365 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.436342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.438408 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.438362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.438581 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.438560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.444677 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.444653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95th8\" (UniqueName: \"kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8\") pod \"auth-test-model-a00003-kserve-869dfbbbbc-dr6ms\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") " pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.523423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.523334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:44.799417 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:44.799385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:53:44.800728 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:53:44.800701    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d538115_0cc7_45c5_ae7e_0770ef3b892d.slice/crio-a747606e6715c1c3c3371218f3c93cbaee082c5a5808eb5b618606ecb7becbce WatchSource:0}: Error finding container a747606e6715c1c3c3371218f3c93cbaee082c5a5808eb5b618606ecb7becbce: Status 404 returned error can't find the container with id a747606e6715c1c3c3371218f3c93cbaee082c5a5808eb5b618606ecb7becbce
Sep 10 13:53:45.755477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:45.755442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" event={"ID":"5d538115-0cc7-45c5-ae7e-0770ef3b892d","Type":"ContainerStarted","Data":"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"}
Sep 10 13:53:45.755477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:45.755484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" event={"ID":"5d538115-0cc7-45c5-ae7e-0770ef3b892d","Type":"ContainerStarted","Data":"a747606e6715c1c3c3371218f3c93cbaee082c5a5808eb5b618606ecb7becbce"}
Sep 10 13:53:45.776041 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:45.775987    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" podStartSLOduration=1.775971776 podStartE2EDuration="1.775971776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:53:45.774454105 +0000 UTC m=+1510.229329270" watchObservedRunningTime="2026-09-10 13:53:45.775971776 +0000 UTC m=+1510.230846952"
Sep 10 13:53:50.243902 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:50.243860    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:53:54.524394 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:54.524358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:54.524787 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:54.524440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:54.536972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:54.536949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:53:54.812696 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:53:54.812608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:54:04.545235 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:04.545198    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:54:20.334083 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:20.334042    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:54:28.985244 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:28.985208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:54:37.833084 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:37.833043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:54:44.005039 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.004966    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" podUID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" containerName="main" containerID="cri-o://c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8" gracePeriod=45
Sep 10 13:54:44.256846 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.256766    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:54:44.330678 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330638    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.330852 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.330852 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.330852 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.331005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vfzn\" (UniqueName: \"kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.331005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.330933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm\") pod \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\" (UID: \"cfcf17e4-a111-44e1-81f9-f5d90e70878c\") "
Sep 10 13:54:44.331101 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.331058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:44.331168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.331087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:44.331373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.331342    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:44.331373 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.331372    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:44.331616 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.331443    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home" (OuterVolumeSpecName: "home") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:44.333195 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.333159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm" (OuterVolumeSpecName: "dshm") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:54:44.333304 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.333244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:54:44.333304 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.333269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn" (OuterVolumeSpecName: "kube-api-access-6vfzn") pod "cfcf17e4-a111-44e1-81f9-f5d90e70878c" (UID: "cfcf17e4-a111-44e1-81f9-f5d90e70878c"). InnerVolumeSpecName "kube-api-access-6vfzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:54:44.432866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.432827    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:44.432866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.432862    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfzn\" (UniqueName: \"kubernetes.io/projected/cfcf17e4-a111-44e1-81f9-f5d90e70878c-kube-api-access-6vfzn\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:44.432866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.432872    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfcf17e4-a111-44e1-81f9-f5d90e70878c-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:44.433140 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:44.432890    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfcf17e4-a111-44e1-81f9-f5d90e70878c-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:54:45.059011 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.058967    2577 generic.go:356] "Generic (PLEG): container finished" podID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" containerID="c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8" exitCode=0
Sep 10 13:54:45.059011 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.059015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"
Sep 10 13:54:45.059601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.059056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" event={"ID":"cfcf17e4-a111-44e1-81f9-f5d90e70878c","Type":"ContainerDied","Data":"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"}
Sep 10 13:54:45.059601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.059105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p" event={"ID":"cfcf17e4-a111-44e1-81f9-f5d90e70878c","Type":"ContainerDied","Data":"45f68ec1d868586608e95e996ff30b8d77feecc4fc4b3fe3d15504147758ea82"}
Sep 10 13:54:45.059601 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.059131    2577 scope.go:117] "RemoveContainer" containerID="c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"
Sep 10 13:54:45.069024 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.068998    2577 scope.go:117] "RemoveContainer" containerID="c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"
Sep 10 13:54:45.069361 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:54:45.069335    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8\": container with ID starting with c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8 not found: ID does not exist" containerID="c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"
Sep 10 13:54:45.069434 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.069376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8"} err="failed to get container status \"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8\": rpc error: code = NotFound desc = could not find container \"c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8\": container with ID starting with c58a15eae8ae24ef62fb57c3cd13ac507c2682554ee861c390116b317ef9a7b8 not found: ID does not exist"
Sep 10 13:54:45.085422 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.085383    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:54:45.090607 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:45.090574    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-713ee90e/e2e-lifecycle-model-713ee90e-kserve-bc4857447-ggw5p"]
Sep 10 13:54:46.063881 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:46.063843    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" path="/var/lib/kubelet/pods/cfcf17e4-a111-44e1-81f9-f5d90e70878c/volumes"
Sep 10 13:54:54.654205 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:54:54.654161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:55:11.660355 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:11.660317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:55:27.241962 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:27.241926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:55:42.030089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:42.030050    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:55:42.922116 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:42.922072    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:55:45.836468 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:45.836425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:55:57.050093 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.050050    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" podUID="fb022141-cfe7-49eb-91a6-c03df264e983" containerName="main" containerID="cri-o://132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67" gracePeriod=45
Sep 10 13:55:57.299386 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.299353    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:55:57.356560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57qpw\" (UniqueName: \"kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356918 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356918 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache\") pod \"fb022141-cfe7-49eb-91a6-c03df264e983\" (UID: \"fb022141-cfe7-49eb-91a6-c03df264e983\") "
Sep 10 13:55:57.356918 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:57.356918 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.356894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home" (OuterVolumeSpecName: "home") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:57.357124 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.357047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:57.357186 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.357117    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.357186 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.357141    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.357186 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.357153    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.358848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.358817    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm" (OuterVolumeSpecName: "dshm") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:57.358964 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.358876    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw" (OuterVolumeSpecName: "kube-api-access-57qpw") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "kube-api-access-57qpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:55:57.359089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.359068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb022141-cfe7-49eb-91a6-c03df264e983" (UID: "fb022141-cfe7-49eb-91a6-c03df264e983"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:55:57.437694 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.437655    2577 generic.go:356] "Generic (PLEG): container finished" podID="fb022141-cfe7-49eb-91a6-c03df264e983" containerID="132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67" exitCode=0
Sep 10 13:55:57.437892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.437701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"
Sep 10 13:55:57.437892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.437709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" event={"ID":"fb022141-cfe7-49eb-91a6-c03df264e983","Type":"ContainerDied","Data":"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"}
Sep 10 13:55:57.437892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.437751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw" event={"ID":"fb022141-cfe7-49eb-91a6-c03df264e983","Type":"ContainerDied","Data":"a5f480d1a10e5b0337c662b7194a005352e42ddadf8ef543a66cfad64228de6d"}
Sep 10 13:55:57.437892 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.437769    2577 scope.go:117] "RemoveContainer" containerID="132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"
Sep 10 13:55:57.447005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.446989    2577 scope.go:117] "RemoveContainer" containerID="132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"
Sep 10 13:55:57.447254 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:55:57.447236    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67\": container with ID starting with 132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67 not found: ID does not exist" containerID="132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"
Sep 10 13:55:57.447325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.447261    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67"} err="failed to get container status \"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67\": rpc error: code = NotFound desc = could not find container \"132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67\": container with ID starting with 132beec74eff22fa7c9fb2a92dccd0ec46b386df28b32d3e715143abf83b8f67 not found: ID does not exist"
Sep 10 13:55:57.458080 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.458050    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57qpw\" (UniqueName: \"kubernetes.io/projected/fb022141-cfe7-49eb-91a6-c03df264e983-kube-api-access-57qpw\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.458080 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.458078    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb022141-cfe7-49eb-91a6-c03df264e983-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.458239 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.458091    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb022141-cfe7-49eb-91a6-c03df264e983-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:57.462100 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.462068    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:55:57.467701 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.467672    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/auth-test-model-7a11e9-kserve-678747fd9f-m64qw"]
Sep 10 13:55:57.942204 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:57.942140    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" podUID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" containerName="main" containerID="cri-o://3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a" gracePeriod=45
Sep 10 13:55:58.068320 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.068279    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb022141-cfe7-49eb-91a6-c03df264e983" path="/var/lib/kubelet/pods/fb022141-cfe7-49eb-91a6-c03df264e983/volumes"
Sep 10 13:55:58.201118 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.201047    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:55:58.265057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265150    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95th8\" (UniqueName: \"kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265259 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm\") pod \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\" (UID: \"5d538115-0cc7-45c5-ae7e-0770ef3b892d\") "
Sep 10 13:55:58.265560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home" (OuterVolumeSpecName: "home") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:58.265560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:58.265560 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:58.265704 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265598    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.265704 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265627    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.265704 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.265636    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.267540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.267486    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm" (OuterVolumeSpecName: "dshm") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:55:58.267540 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.267489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:55:58.267705 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.267621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8" (OuterVolumeSpecName: "kube-api-access-95th8") pod "5d538115-0cc7-45c5-ae7e-0770ef3b892d" (UID: "5d538115-0cc7-45c5-ae7e-0770ef3b892d"). InnerVolumeSpecName "kube-api-access-95th8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:55:58.366856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.366815    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95th8\" (UniqueName: \"kubernetes.io/projected/5d538115-0cc7-45c5-ae7e-0770ef3b892d-kube-api-access-95th8\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.366856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.366847    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d538115-0cc7-45c5-ae7e-0770ef3b892d-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.366856 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.366856    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d538115-0cc7-45c5-ae7e-0770ef3b892d-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:55:58.445561 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.445521    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" containerID="3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a" exitCode=0
Sep 10 13:55:58.445753 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.445572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"
Sep 10 13:55:58.445753 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.445581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" event={"ID":"5d538115-0cc7-45c5-ae7e-0770ef3b892d","Type":"ContainerDied","Data":"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"}
Sep 10 13:55:58.445753 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.445619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms" event={"ID":"5d538115-0cc7-45c5-ae7e-0770ef3b892d","Type":"ContainerDied","Data":"a747606e6715c1c3c3371218f3c93cbaee082c5a5808eb5b618606ecb7becbce"}
Sep 10 13:55:58.445753 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.445641    2577 scope.go:117] "RemoveContainer" containerID="3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"
Sep 10 13:55:58.454899 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.454874    2577 scope.go:117] "RemoveContainer" containerID="3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"
Sep 10 13:55:58.455234 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:55:58.455212    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a\": container with ID starting with 3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a not found: ID does not exist" containerID="3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"
Sep 10 13:55:58.455322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.455248    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a"} err="failed to get container status \"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a\": rpc error: code = NotFound desc = could not find container \"3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a\": container with ID starting with 3a585175d62e06d644491d890be3adb704bda08b59871eaa50acf1cd3d1b576a not found: ID does not exist"
Sep 10 13:55:58.468688 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.468656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:55:58.473044 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.473019    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/auth-test-model-a00003-kserve-869dfbbbbc-dr6ms"]
Sep 10 13:55:58.847027 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:55:58.846943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:56:00.064163 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:00.064131    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" path="/var/lib/kubelet/pods/5d538115-0cc7-45c5-ae7e-0770ef3b892d/volumes"
Sep 10 13:56:25.011197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.011158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:56:25.012527 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" containerName="main"
Sep 10 13:56:25.012527 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012524    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" containerName="main"
Sep 10 13:56:25.012707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012550    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb022141-cfe7-49eb-91a6-c03df264e983" containerName="main"
Sep 10 13:56:25.012707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb022141-cfe7-49eb-91a6-c03df264e983" containerName="main"
Sep 10 13:56:25.012707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012588    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" containerName="main"
Sep 10 13:56:25.012707 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" containerName="main"
Sep 10 13:56:25.012840 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012715    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d538115-0cc7-45c5-ae7e-0770ef3b892d" containerName="main"
Sep 10 13:56:25.012840 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012731    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb022141-cfe7-49eb-91a6-c03df264e983" containerName="main"
Sep 10 13:56:25.012840 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.012743    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfcf17e4-a111-44e1-81f9-f5d90e70878c" containerName="main"
Sep 10 13:56:25.016280 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.016258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.018719 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.018693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ab74b3-dockercfg-dg7kj\""
Sep 10 13:56:25.018928 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.018907    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ab74b3\""
Sep 10 13:56:25.025033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.025006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:56:25.124888 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.124836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59sw\" (UniqueName: \"kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.125081 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.124912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.225546 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.225471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.225756 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.225655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59sw\" (UniqueName: \"kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.226274 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.226251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.234740 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.234705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59sw\" (UniqueName: \"kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw\") pod \"payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.328593 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.328443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:25.606788 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.606758    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:56:25.609059 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:56:25.609027    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9834c462_a0d7_4305_b648_f246273244bf.slice/crio-fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37 WatchSource:0}: Error finding container fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37: Status 404 returned error can't find the container with id fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37
Sep 10 13:56:25.611141 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:25.611118    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:56:26.593207 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:26.593151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" event={"ID":"9834c462-a0d7-4305-b648-f246273244bf","Type":"ContainerStarted","Data":"fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37"}
Sep 10 13:56:28.605806 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:28.605768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" event={"ID":"9834c462-a0d7-4305-b648-f246273244bf","Type":"ContainerStarted","Data":"947f1a12591b5f3bf54cebf73cde0736ede9a357cf8d75bbb810784c11b86ce6"}
Sep 10 13:56:28.606206 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:28.605842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:28.625522 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:28.625464    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" podStartSLOduration=2.17476849 podStartE2EDuration="4.625444134s" totalImagesPullingTime="2.450675644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:28.624384769 +0000 UTC m=+1673.079259922" watchObservedRunningTime="2026-09-10 13:56:28.625444134 +0000 UTC m=+1673.080319297"
Sep 10 13:56:29.358105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:29.358061    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:56:36.808087 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.808048    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:56:36.812956 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.812935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.826735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.826710    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"kube-root-ca.crt\""
Sep 10 13:56:36.826872 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.826710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"sub-test-model-a00003-kserve-self-signed-certs\""
Sep 10 13:56:36.826872 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.826718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"default-dockercfg-4cvkz\""
Sep 10 13:56:36.829323 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.829301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"openshift-service-ca.crt\""
Sep 10 13:56:36.838562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.838531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:56:36.842496 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.842640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.842640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.842640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.842780 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779bf\" (UniqueName: \"kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.842780 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.842742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944063 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779bf\" (UniqueName: \"kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944541 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944615 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.944773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.944754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.946325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.946304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.946608 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.946588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:36.958199 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:36.958172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779bf\" (UniqueName: \"kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf\") pod \"sub-test-model-a00003-kserve-87556bcbc-lzvl9\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") " pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:37.122468 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:37.122423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:37.410460 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:37.410429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:56:37.412050 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:56:37.412024    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bb3380_de83_4849_8221_0ed549057946.slice/crio-97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd WatchSource:0}: Error finding container 97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd: Status 404 returned error can't find the container with id 97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd
Sep 10 13:56:37.653340 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:37.653297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" event={"ID":"72bb3380-de83-4849-8221-0ed549057946","Type":"ContainerStarted","Data":"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"}
Sep 10 13:56:37.653612 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:37.653348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" event={"ID":"72bb3380-de83-4849-8221-0ed549057946","Type":"ContainerStarted","Data":"97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd"}
Sep 10 13:56:37.673123 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:37.673002    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" podStartSLOduration=1.672983961 podStartE2EDuration="1.672983961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:37.671041001 +0000 UTC m=+1682.125916176" watchObservedRunningTime="2026-09-10 13:56:37.672983961 +0000 UTC m=+1682.127859125"
Sep 10 13:56:39.613914 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:39.613875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:56:41.085800 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:41.085762    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:56:47.122839 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:47.122797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:47.125146 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:47.123108    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:47.136061 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:47.136030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:47.715562 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:47.715504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:56:53.552292 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.552258    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:56:53.555854 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.555836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.560586 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.560560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8aa282-openshift-default-dockercfg-6tjdq\""
Sep 10 13:56:53.589414 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.589382    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:56:53.615354 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjpr\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615584 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615677 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615727 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615832 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615939 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.615939 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.615888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717576 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717772 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.717866 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.717981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjpr\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.718055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.718149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.718302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718634 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.718608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.718736 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.718717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.719939 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.719922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.720253 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.720232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.727266 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.727237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.727440 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.727408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjpr\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr\") pod \"e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") " pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:53.866350 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:53.866322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:54.146944 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.146915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:56:54.148780 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:56:54.148737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e44e833_3bfb_4537_91e1_ac799630c530.slice/crio-d7b61b1585ee92c3f081715be8803aa6fa9cc72ebf83b3a341e716bd9e0772e4 WatchSource:0}: Error finding container d7b61b1585ee92c3f081715be8803aa6fa9cc72ebf83b3a341e716bd9e0772e4: Status 404 returned error can't find the container with id d7b61b1585ee92c3f081715be8803aa6fa9cc72ebf83b3a341e716bd9e0772e4
Sep 10 13:56:54.151227 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.151190    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:56:54.151330 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.151307    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:56:54.151393 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.151356    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:56:54.741908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.741868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerStarted","Data":"ebd94f4bc3d2ca7f0c8cc172635481898bfd49ece103f088c8a15526bf808da9"}
Sep 10 13:56:54.742289 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.741935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerStarted","Data":"d7b61b1585ee92c3f081715be8803aa6fa9cc72ebf83b3a341e716bd9e0772e4"}
Sep 10 13:56:54.768662 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.768609    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podStartSLOduration=1.768592 podStartE2EDuration="1.768592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:54.766752269 +0000 UTC m=+1699.221627433" watchObservedRunningTime="2026-09-10 13:56:54.768592 +0000 UTC m=+1699.223467163"
Sep 10 13:56:54.867410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:54.867375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:55.872028 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:55.871990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:56.753302 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:56.753268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:56.754423 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:56.754401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:56:58.891297 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.891215    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:56:58.898104 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.898080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:58.900969 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.900947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8aa282-dockercfg-bz746\""
Sep 10 13:56:58.900969 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.900967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8aa282\""
Sep 10 13:56:58.909453 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.909429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:56:58.973257 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.973217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wmw\" (UniqueName: \"kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:58.973448 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:58.973285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.074212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.074172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.074418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.074307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wmw\" (UniqueName: \"kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.074955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.074932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.085982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.085960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wmw\" (UniqueName: \"kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw\") pod \"payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.209646 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.209560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.486865 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.486837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:56:59.488662 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:56:59.488631    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ace8ed_bdf9_45eb_a09c_2d4452f766ac.slice/crio-4bf329f512dbe55f3ca744bd6192337a3090e7801f02270cb1b5ade23cfc7ffd WatchSource:0}: Error finding container 4bf329f512dbe55f3ca744bd6192337a3090e7801f02270cb1b5ade23cfc7ffd: Status 404 returned error can't find the container with id 4bf329f512dbe55f3ca744bd6192337a3090e7801f02270cb1b5ade23cfc7ffd
Sep 10 13:56:59.775129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.775020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" event={"ID":"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac","Type":"ContainerStarted","Data":"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"}
Sep 10 13:56:59.775129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.775085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" event={"ID":"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac","Type":"ContainerStarted","Data":"4bf329f512dbe55f3ca744bd6192337a3090e7801f02270cb1b5ade23cfc7ffd"}
Sep 10 13:56:59.775325 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.775199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:56:59.800785 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:56:59.800730    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" podStartSLOduration=1.8007144579999999 podStartE2EDuration="1.800714458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:59.797431343 +0000 UTC m=+1704.252306506" watchObservedRunningTime="2026-09-10 13:56:59.800714458 +0000 UTC m=+1704.255589623"
Sep 10 13:57:05.848982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:05.848942    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:57:10.783669 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:10.783619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:57:16.880151 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.880075    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e44e833-3bfb-4537-91e1-ac799630c530" containerID="ebd94f4bc3d2ca7f0c8cc172635481898bfd49ece103f088c8a15526bf808da9" exitCode=0
Sep 10 13:57:16.880572 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.880147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerDied","Data":"ebd94f4bc3d2ca7f0c8cc172635481898bfd49ece103f088c8a15526bf808da9"}
Sep 10 13:57:16.880640 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.880625    2577 scope.go:117] "RemoveContainer" containerID="ebd94f4bc3d2ca7f0c8cc172635481898bfd49ece103f088c8a15526bf808da9"
Sep 10 13:57:16.881771 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.881737    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:57:16.881843 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.881811    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:57:16.881888 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:16.881851    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:57:17.887410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:17.887374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerStarted","Data":"7aaa92c8430e167e2557dbe43926dc8fdda532ce66bf25668619c13340c125a6"}
Sep 10 13:57:18.867206 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:18.867156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:57:18.867412 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:18.867391    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 13:57:18.867496 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:18.867467    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 13:57:19.867558 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:19.867486    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 13:57:19.867951 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:19.867593    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 13:57:20.867485 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:20.867442    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 13:57:20.867702 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:20.867538    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 13:57:21.866727 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:21.866689    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 13:57:21.866964 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:21.866943    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 13:57:22.867436 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:22.867392    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 10 13:57:22.867937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:22.867462    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 10 13:57:23.962077 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:23.962021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:57:23.962636 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:23.962610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:57:23.964336 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:23.964310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:57:24.949546 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:24.949492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:57:43.054735 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:43.054695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:57:45.769828 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:45.769790    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:57:45.770319 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:45.770015    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" podUID="9834c462-a0d7-4305-b648-f246273244bf" containerName="payload-pre-processing" containerID="cri-o://947f1a12591b5f3bf54cebf73cde0736ede9a357cf8d75bbb810784c11b86ce6" gracePeriod=30
Sep 10 13:57:46.036486 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.036445    2577 generic.go:356] "Generic (PLEG): container finished" podID="9834c462-a0d7-4305-b648-f246273244bf" containerID="947f1a12591b5f3bf54cebf73cde0736ede9a357cf8d75bbb810784c11b86ce6" exitCode=0
Sep 10 13:57:46.036668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.036534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" event={"ID":"9834c462-a0d7-4305-b648-f246273244bf","Type":"ContainerDied","Data":"947f1a12591b5f3bf54cebf73cde0736ede9a357cf8d75bbb810784c11b86ce6"}
Sep 10 13:57:46.036668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.036634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg" event={"ID":"9834c462-a0d7-4305-b648-f246273244bf","Type":"ContainerDied","Data":"fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37"}
Sep 10 13:57:46.036668 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.036658    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6a2222937e2285df24b848ae9be63bdacdd8233b2e0a266f8dcaaad4c94f37"
Sep 10 13:57:46.047306 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.047282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:57:46.136728 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.136686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume\") pod \"9834c462-a0d7-4305-b648-f246273244bf\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") "
Sep 10 13:57:46.136894 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.136793    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h59sw\" (UniqueName: \"kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw\") pod \"9834c462-a0d7-4305-b648-f246273244bf\" (UID: \"9834c462-a0d7-4305-b648-f246273244bf\") "
Sep 10 13:57:46.137184 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.137157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9834c462-a0d7-4305-b648-f246273244bf" (UID: "9834c462-a0d7-4305-b648-f246273244bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:57:46.138990 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.138964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw" (OuterVolumeSpecName: "kube-api-access-h59sw") pod "9834c462-a0d7-4305-b648-f246273244bf" (UID: "9834c462-a0d7-4305-b648-f246273244bf"). InnerVolumeSpecName "kube-api-access-h59sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:57:46.238836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.238795    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59sw\" (UniqueName: \"kubernetes.io/projected/9834c462-a0d7-4305-b648-f246273244bf-kube-api-access-h59sw\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:57:46.238836 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:46.238831    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9834c462-a0d7-4305-b648-f246273244bf-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:57:47.041743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:47.041708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"
Sep 10 13:57:47.102859 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:47.102828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:57:47.109151 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:47.109122    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ab74b3-8f679b5cd-4kbzg"]
Sep 10 13:57:48.063751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:48.063716    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9834c462-a0d7-4305-b648-f246273244bf" path="/var/lib/kubelet/pods/9834c462-a0d7-4305-b648-f246273244bf/volumes"
Sep 10 13:57:53.324103 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:53.324055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:57:58.155915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:57:58.155882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:58:08.348494 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.348451    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" podUID="72bb3380-de83-4849-8221-0ed549057946" containerName="main" containerID="cri-o://93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42" gracePeriod=45
Sep 10 13:58:08.607775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.607705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:58:08.656218 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656387 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-779bf\" (UniqueName: \"kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs\") pod \"72bb3380-de83-4849-8221-0ed549057946\" (UID: \"72bb3380-de83-4849-8221-0ed549057946\") "
Sep 10 13:58:08.656630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache" (OuterVolumeSpecName: "model-cache") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:08.656630 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:08.656847 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home" (OuterVolumeSpecName: "home") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:08.656959 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656938    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:08.657024 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656965    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:08.657024 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.656982    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:08.658587 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.658559    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm" (OuterVolumeSpecName: "dshm") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:08.658974 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.658953    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:58:08.659055 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.658968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf" (OuterVolumeSpecName: "kube-api-access-779bf") pod "72bb3380-de83-4849-8221-0ed549057946" (UID: "72bb3380-de83-4849-8221-0ed549057946"). InnerVolumeSpecName "kube-api-access-779bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:08.757941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.757896    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72bb3380-de83-4849-8221-0ed549057946-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:08.757941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.757930    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72bb3380-de83-4849-8221-0ed549057946-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:08.757941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:08.757944    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779bf\" (UniqueName: \"kubernetes.io/projected/72bb3380-de83-4849-8221-0ed549057946-kube-api-access-779bf\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:09.156382 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.156342    2577 generic.go:356] "Generic (PLEG): container finished" podID="72bb3380-de83-4849-8221-0ed549057946" containerID="93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42" exitCode=0
Sep 10 13:58:09.156588 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.156407    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"
Sep 10 13:58:09.156588 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.156421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" event={"ID":"72bb3380-de83-4849-8221-0ed549057946","Type":"ContainerDied","Data":"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"}
Sep 10 13:58:09.156588 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.156493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9" event={"ID":"72bb3380-de83-4849-8221-0ed549057946","Type":"ContainerDied","Data":"97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd"}
Sep 10 13:58:09.156588 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.156529    2577 scope.go:117] "RemoveContainer" containerID="93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"
Sep 10 13:58:09.166769 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.166747    2577 scope.go:117] "RemoveContainer" containerID="93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"
Sep 10 13:58:09.167079 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:09.167056    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42\": container with ID starting with 93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42 not found: ID does not exist" containerID="93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"
Sep 10 13:58:09.167136 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.167089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42"} err="failed to get container status \"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42\": rpc error: code = NotFound desc = could not find container \"93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42\": container with ID starting with 93ec2f6fb40acc2aef6db162ed07a334276a4ae685650a4f568c6f67b08c1f42 not found: ID does not exist"
Sep 10 13:58:09.192655 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.192613    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:58:09.194210 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:09.194170    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bb3380_de83_4849_8221_0ed549057946.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72bb3380_de83_4849_8221_0ed549057946.slice/crio-97997324f4f4b373d93c72bde3ea3547279a53c283dce02273302347548c55dd\": RecentStats: unable to find data in memory cache]"
Sep 10 13:58:09.198168 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:09.198132    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/sub-test-model-a00003-kserve-87556bcbc-lzvl9"]
Sep 10 13:58:10.063770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.063732    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72bb3380-de83-4849-8221-0ed549057946" path="/var/lib/kubelet/pods/72bb3380-de83-4849-8221-0ed549057946/volumes"
Sep 10 13:58:10.526239 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526203    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:58:10.526651 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526637    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72bb3380-de83-4849-8221-0ed549057946" containerName="main"
Sep 10 13:58:10.526709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526667    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="72bb3380-de83-4849-8221-0ed549057946" containerName="main"
Sep 10 13:58:10.526709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526687    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9834c462-a0d7-4305-b648-f246273244bf" containerName="payload-pre-processing"
Sep 10 13:58:10.526709 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9834c462-a0d7-4305-b648-f246273244bf" containerName="payload-pre-processing"
Sep 10 13:58:10.526814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526765    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="72bb3380-de83-4849-8221-0ed549057946" containerName="main"
Sep 10 13:58:10.526814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.526776    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9834c462-a0d7-4305-b648-f246273244bf" containerName="payload-pre-processing"
Sep 10 13:58:10.531099 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.531081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.538299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.538272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"default-dockercfg-2jqwd\""
Sep 10 13:58:10.538435 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.538305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"kube-root-ca.crt\""
Sep 10 13:58:10.538435 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.538347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"openshift-service-ca.crt\""
Sep 10 13:58:10.538567 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.538444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7a11e9\"/\"rate-test-model-7a11e9-kserve-self-signed-certs\""
Sep 10 13:58:10.550525 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.550483    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:58:10.575062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dwst\" (UniqueName: \"kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.575258 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.575258 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.575258 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.575388 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.575388 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.575312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676564 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dwst\" (UniqueName: \"kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676755 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676815 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676815 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.676935 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.676932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.677078 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.677053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.677225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.677197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.677319 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.677235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.679074 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.679050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.679237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.679221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.692590 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.692565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dwst\" (UniqueName: \"kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst\") pod \"rate-test-model-7a11e9-kserve-5b84c86cf-pm82g\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") " pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:10.841352 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:10.841263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:11.056271 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.056239    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:58:11.056470 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.056449    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" podUID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" containerName="payload-pre-processing" containerID="cri-o://b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99" gracePeriod=30
Sep 10 13:58:11.117088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.117050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:58:11.128802 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:58:11.128765    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ec080e0_61b2_4847_b47b_6d09ad342d0a.slice/crio-85b10bb804649bb5d29581cdc4a8e510a6d5f453312fde6fa04788dbcc7a30ca WatchSource:0}: Error finding container 85b10bb804649bb5d29581cdc4a8e510a6d5f453312fde6fa04788dbcc7a30ca: Status 404 returned error can't find the container with id 85b10bb804649bb5d29581cdc4a8e510a6d5f453312fde6fa04788dbcc7a30ca
Sep 10 13:58:11.171237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.171203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" event={"ID":"2ec080e0-61b2-4847-b47b-6d09ad342d0a","Type":"ContainerStarted","Data":"85b10bb804649bb5d29581cdc4a8e510a6d5f453312fde6fa04788dbcc7a30ca"}
Sep 10 13:58:11.306042 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.306016    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:58:11.383834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.383727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume\") pod \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") "
Sep 10 13:58:11.383834 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.383827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8wmw\" (UniqueName: \"kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw\") pod \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\" (UID: \"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac\") "
Sep 10 13:58:11.384193 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.384161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" (UID: "a8ace8ed-bdf9-45eb-a09c-2d4452f766ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:58:11.386608 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.386567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw" (OuterVolumeSpecName: "kube-api-access-m8wmw") pod "a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" (UID: "a8ace8ed-bdf9-45eb-a09c-2d4452f766ac"). InnerVolumeSpecName "kube-api-access-m8wmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:11.484972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.484930    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:11.484972 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:11.484965    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wmw\" (UniqueName: \"kubernetes.io/projected/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac-kube-api-access-m8wmw\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:12.178405 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.178371    2577 generic.go:356] "Generic (PLEG): container finished" podID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" containerID="b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99" exitCode=0
Sep 10 13:58:12.178908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.178421    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"
Sep 10 13:58:12.178908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.178458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" event={"ID":"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac","Type":"ContainerDied","Data":"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"}
Sep 10 13:58:12.178908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.178502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh" event={"ID":"a8ace8ed-bdf9-45eb-a09c-2d4452f766ac","Type":"ContainerDied","Data":"4bf329f512dbe55f3ca744bd6192337a3090e7801f02270cb1b5ade23cfc7ffd"}
Sep 10 13:58:12.178908 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.178541    2577 scope.go:117] "RemoveContainer" containerID="b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"
Sep 10 13:58:12.180573 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.180548    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" event={"ID":"2ec080e0-61b2-4847-b47b-6d09ad342d0a","Type":"ContainerStarted","Data":"aee9f7c653e45ce1d5a1269e5f86d366c77785fcc9a7043b5189ed7bff49e4ba"}
Sep 10 13:58:12.187803 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.187781    2577 scope.go:117] "RemoveContainer" containerID="b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"
Sep 10 13:58:12.194036 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:12.188659    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99\": container with ID starting with b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99 not found: ID does not exist" containerID="b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"
Sep 10 13:58:12.194036 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.188717    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99"} err="failed to get container status \"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99\": rpc error: code = NotFound desc = could not find container \"b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99\": container with ID starting with b605ab56a240f6db777fd3ff60db2088d93a4703f01ed5459877a866a7f80e99 not found: ID does not exist"
Sep 10 13:58:12.208447 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.208409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:58:12.218955 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:12.218925    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8aa282-554ddd4f9d-v78sh"]
Sep 10 13:58:14.064321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:14.064286    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" path="/var/lib/kubelet/pods/a8ace8ed-bdf9-45eb-a09c-2d4452f766ac/volumes"
Sep 10 13:58:14.929759 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:14.929696    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" podStartSLOduration=4.929675033 podStartE2EDuration="4.929675033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:12.268198828 +0000 UTC m=+1776.723073991" watchObservedRunningTime="2026-09-10 13:58:14.929675033 +0000 UTC m=+1779.384550197"
Sep 10 13:58:14.929959 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:14.929843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:58:19.046429 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:19.046393    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:58:19.046958 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:19.046695    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" containerID="cri-o://7aaa92c8430e167e2557dbe43926dc8fdda532ce66bf25668619c13340c125a6" gracePeriod=30
Sep 10 13:58:20.841419 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:20.841380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:20.841982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:20.841444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:20.854631 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:20.854597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:21.241310 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:21.241266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:58:23.965409 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:23.965366    2577 patch_prober.go:28] interesting pod/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:58:23.965810 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:23.965437    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:58:24.247917 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.247836    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e44e833-3bfb-4537-91e1-ac799630c530" containerID="7aaa92c8430e167e2557dbe43926dc8fdda532ce66bf25668619c13340c125a6" exitCode=0
Sep 10 13:58:24.247917 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.247906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerDied","Data":"7aaa92c8430e167e2557dbe43926dc8fdda532ce66bf25668619c13340c125a6"}
Sep 10 13:58:24.248105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.247987    2577 scope.go:117] "RemoveContainer" containerID="ebd94f4bc3d2ca7f0c8cc172635481898bfd49ece103f088c8a15526bf808da9"
Sep 10 13:58:24.293585 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.293557    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:58:24.419413 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tjpr\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419413 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419557    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.419721 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data\") pod \"3e44e833-3bfb-4537-91e1-ac799630c530\" (UID: \"3e44e833-3bfb-4537-91e1-ac799630c530\") "
Sep 10 13:58:24.420088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:24.420088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:24.420088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.419986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:24.420088 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:24.420299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420208    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.420299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420229    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.420299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420244    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.420299 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420257    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.420527 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.420313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:58:24.422031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.421999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:24.422031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.422011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:58:24.422189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.422026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr" (OuterVolumeSpecName: "kube-api-access-6tjpr") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "kube-api-access-6tjpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:24.422189 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.422058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e44e833-3bfb-4537-91e1-ac799630c530" (UID: "3e44e833-3bfb-4537-91e1-ac799630c530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:24.521775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.521682    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.521775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.521717    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e44e833-3bfb-4537-91e1-ac799630c530-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.521775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.521727    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e44e833-3bfb-4537-91e1-ac799630c530-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.521775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.521735    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tjpr\" (UniqueName: \"kubernetes.io/projected/3e44e833-3bfb-4537-91e1-ac799630c530-kube-api-access-6tjpr\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:24.521775 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:24.521746    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e44e833-3bfb-4537-91e1-ac799630c530-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:25.262388 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:25.262354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc" event={"ID":"3e44e833-3bfb-4537-91e1-ac799630c530","Type":"ContainerDied","Data":"d7b61b1585ee92c3f081715be8803aa6fa9cc72ebf83b3a341e716bd9e0772e4"}
Sep 10 13:58:25.262887 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:25.262394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"
Sep 10 13:58:25.262887 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:25.262438    2577 scope.go:117] "RemoveContainer" containerID="7aaa92c8430e167e2557dbe43926dc8fdda532ce66bf25668619c13340c125a6"
Sep 10 13:58:25.287978 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:25.287939    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:58:25.295862 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:25.295830    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8aa282-openshift-default-64c4fdc946-2hvgc"]
Sep 10 13:58:26.064530 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:26.064475    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" path="/var/lib/kubelet/pods/3e44e833-3bfb-4537-91e1-ac799630c530/volumes"
Sep 10 13:58:30.014928 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.014851    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:58:30.015402 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.015080    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" podUID="3ac798a4-d360-4bd1-84c8-591a37e941e8" containerName="maas-api" containerID="cri-o://49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15" gracePeriod=30
Sep 10 13:58:30.273825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.273763    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:58:30.289900 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.289865    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ac798a4-d360-4bd1-84c8-591a37e941e8" containerID="49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15" exitCode=0
Sep 10 13:58:30.290057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.289919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"
Sep 10 13:58:30.290057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.289947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" event={"ID":"3ac798a4-d360-4bd1-84c8-591a37e941e8","Type":"ContainerDied","Data":"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"}
Sep 10 13:58:30.290057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.290000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb" event={"ID":"3ac798a4-d360-4bd1-84c8-591a37e941e8","Type":"ContainerDied","Data":"45d792a6b7886dc02f0a82f9792219e4befa3eb003d9c29301d3a249726f743a"}
Sep 10 13:58:30.290057 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.290023    2577 scope.go:117] "RemoveContainer" containerID="49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"
Sep 10 13:58:30.301483 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.301241    2577 scope.go:117] "RemoveContainer" containerID="49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"
Sep 10 13:58:30.301709 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:30.301679    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15\": container with ID starting with 49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15 not found: ID does not exist" containerID="49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"
Sep 10 13:58:30.301825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.301721    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15"} err="failed to get container status \"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15\": rpc error: code = NotFound desc = could not find container \"49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15\": container with ID starting with 49c701acab130c05f4bee6458e310a85b2b558b0b0d222c089c7e9011ab2ad15 not found: ID does not exist"
Sep 10 13:58:30.381750 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.381712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") pod \"3ac798a4-d360-4bd1-84c8-591a37e941e8\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") "
Sep 10 13:58:30.381937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.381785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca\") pod \"3ac798a4-d360-4bd1-84c8-591a37e941e8\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") "
Sep 10 13:58:30.381937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.381812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gqmd\" (UniqueName: \"kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd\") pod \"3ac798a4-d360-4bd1-84c8-591a37e941e8\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") "
Sep 10 13:58:30.381937 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.381844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs\") pod \"3ac798a4-d360-4bd1-84c8-591a37e941e8\" (UID: \"3ac798a4-d360-4bd1-84c8-591a37e941e8\") "
Sep 10 13:58:30.382212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.382183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ac798a4-d360-4bd1-84c8-591a37e941e8" (UID: "3ac798a4-d360-4bd1-84c8-591a37e941e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:58:30.383992 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.383964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ac798a4-d360-4bd1-84c8-591a37e941e8" (UID: "3ac798a4-d360-4bd1-84c8-591a37e941e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:58:30.384105 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.384033    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3ac798a4-d360-4bd1-84c8-591a37e941e8" (UID: "3ac798a4-d360-4bd1-84c8-591a37e941e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:58:30.384155 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.384109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd" (OuterVolumeSpecName: "kube-api-access-2gqmd") pod "3ac798a4-d360-4bd1-84c8-591a37e941e8" (UID: "3ac798a4-d360-4bd1-84c8-591a37e941e8"). InnerVolumeSpecName "kube-api-access-2gqmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:30.483582 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.483547    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:30.483582 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.483573    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ac798a4-d360-4bd1-84c8-591a37e941e8-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:30.483582 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.483583    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac798a4-d360-4bd1-84c8-591a37e941e8-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:30.483825 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.483594    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqmd\" (UniqueName: \"kubernetes.io/projected/3ac798a4-d360-4bd1-84c8-591a37e941e8-kube-api-access-2gqmd\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:30.618304 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.618269    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:58:30.621773 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:30.621746    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-83c318-5594c5dd7c-fjlmb"]
Sep 10 13:58:31.929570 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929531    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:58:31.929961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929928    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac798a4-d360-4bd1-84c8-591a37e941e8" containerName="maas-api"
Sep 10 13:58:31.929961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929941    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac798a4-d360-4bd1-84c8-591a37e941e8" containerName="maas-api"
Sep 10 13:58:31.929961 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929953    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:58:31.930070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929961    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:58:31.930070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929975    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" containerName="payload-pre-processing"
Sep 10 13:58:31.930070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.929982    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" containerName="payload-pre-processing"
Sep 10 13:58:31.930070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.930059    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac798a4-d360-4bd1-84c8-591a37e941e8" containerName="maas-api"
Sep 10 13:58:31.930070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.930065    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:58:31.930225 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.930073    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8ace8ed-bdf9-45eb-a09c-2d4452f766ac" containerName="payload-pre-processing"
Sep 10 13:58:31.934848 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.934827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.939321 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.939293    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"kube-root-ca.crt\""
Sep 10 13:58:31.940199 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.940171    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"default-dockercfg-4cvkz\""
Sep 10 13:58:31.940322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.940178    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"rate-test-model-a00003-kserve-self-signed-certs\""
Sep 10 13:58:31.940322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.940292    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a00003\"/\"openshift-service-ca.crt\""
Sep 10 13:58:31.947324 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.947303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:58:31.998054 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.998054 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.998260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.998260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.998260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:31.998260 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:31.998169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2tjw\" (UniqueName: \"kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.065062 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.065029    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac798a4-d360-4bd1-84c8-591a37e941e8" path="/var/lib/kubelet/pods/3ac798a4-d360-4bd1-84c8-591a37e941e8/volumes"
Sep 10 13:58:32.098793 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.098763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.098967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.098811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tjw\" (UniqueName: \"kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.098967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.098887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.098967 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.098906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.099127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.099032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.099127 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.099078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.099290 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.099268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.099330 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.099276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.099380 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.099360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.101144 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.101119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.101370 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.101351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.108040 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.108020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tjw\" (UniqueName: \"kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw\") pod \"rate-test-model-a00003-kserve-7856fc8dc-hnk6c\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") " pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.244639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.244554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:32.527266 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:32.527240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:58:32.529382 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:58:32.529350    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7d01101_ac6f_4523_99f0_8fb3907627b7.slice/crio-0be06c133fb420af51cd0cfdbf4c5aea990835c6b62e93585231b281be3005e0 WatchSource:0}: Error finding container 0be06c133fb420af51cd0cfdbf4c5aea990835c6b62e93585231b281be3005e0: Status 404 returned error can't find the container with id 0be06c133fb420af51cd0cfdbf4c5aea990835c6b62e93585231b281be3005e0
Sep 10 13:58:33.311095 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:33.311050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" event={"ID":"f7d01101-ac6f-4523-99f0-8fb3907627b7","Type":"ContainerStarted","Data":"e1c2d8b3a50205455dbb6e2688deef4bc3bb8c776311a9bcc4692c6b91b603d9"}
Sep 10 13:58:33.311491 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:33.311106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" event={"ID":"f7d01101-ac6f-4523-99f0-8fb3907627b7","Type":"ContainerStarted","Data":"0be06c133fb420af51cd0cfdbf4c5aea990835c6b62e93585231b281be3005e0"}
Sep 10 13:58:33.331046 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:33.330992    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" podStartSLOduration=2.330975087 podStartE2EDuration="2.330975087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:58:33.328989623 +0000 UTC m=+1797.783864785" watchObservedRunningTime="2026-09-10 13:58:33.330975087 +0000 UTC m=+1797.785850249"
Sep 10 13:58:35.653765 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:35.653720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:58:36.142045 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:36.142014    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:58:36.144279 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:36.144259    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 13:58:38.386814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:38.386780    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:58:38.387308 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:38.387001    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy" containerID="cri-o://d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb" gracePeriod=30
Sep 10 13:58:42.245636 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:42.245598    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:42.245636 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:42.245647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:42.258349 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:42.258320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:42.367863 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:42.367827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:58:43.648763 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.648733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:58:43.721031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.720932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.720982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k999p\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721031 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721375 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721363    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs\") pod \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\" (UID: \"3c9f321f-c6be-4cd9-9454-b20c4436ce9f\") "
Sep 10 13:58:43.721799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:43.721799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721768    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:43.721799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:58:43.721938 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721772    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.721938 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721822    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.721938 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.721838    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.723435 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.723408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p" (OuterVolumeSpecName: "kube-api-access-k999p") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "kube-api-access-k999p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:43.723435 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.723426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:43.723718 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.723701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:58:43.723770 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.723736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c9f321f-c6be-4cd9-9454-b20c4436ce9f" (UID: "3c9f321f-c6be-4cd9-9454-b20c4436ce9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:58:43.822625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822586    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.822625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822617    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k999p\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-kube-api-access-k999p\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.822625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822628    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.822625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822636    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.822896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822644    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:43.822896 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:43.822653    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9f321f-c6be-4cd9-9454-b20c4436ce9f-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:44.368235 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.368199    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerID="d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb" exitCode=0
Sep 10 13:58:44.368426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.368240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerDied","Data":"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"}
Sep 10 13:58:44.368426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.368269    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"
Sep 10 13:58:44.368426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.368301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz" event={"ID":"3c9f321f-c6be-4cd9-9454-b20c4436ce9f","Type":"ContainerDied","Data":"ec32cf3c02d78ef1d3c15e07e2db5c138a666d72c9257eeec4d94a528109cbf2"}
Sep 10 13:58:44.368426 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.368329    2577 scope.go:117] "RemoveContainer" containerID="d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"
Sep 10 13:58:44.377766 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.377735    2577 scope.go:117] "RemoveContainer" containerID="5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"
Sep 10 13:58:44.387532 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.387482    2577 scope.go:117] "RemoveContainer" containerID="d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"
Sep 10 13:58:44.387804 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:44.387784    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb\": container with ID starting with d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb not found: ID does not exist" containerID="d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"
Sep 10 13:58:44.387874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.387814    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb"} err="failed to get container status \"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb\": rpc error: code = NotFound desc = could not find container \"d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb\": container with ID starting with d7a69e57efd530d9e64c91839adc1eff4d1cb18ab60f1440d5263e88d0a4d8eb not found: ID does not exist"
Sep 10 13:58:44.387874 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.387833    2577 scope.go:117] "RemoveContainer" containerID="5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"
Sep 10 13:58:44.388091 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:44.388071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7\": container with ID starting with 5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7 not found: ID does not exist" containerID="5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"
Sep 10 13:58:44.388133 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.388097    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7"} err="failed to get container status \"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7\": rpc error: code = NotFound desc = could not find container \"5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7\": container with ID starting with 5d5f491a54ad10066e99a8487e77aa86bd3a751eed5fabc11198a416e43e64f7 not found: ID does not exist"
Sep 10 13:58:44.402098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.402071    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:58:44.413064 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:44.413028    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-83c318-openshift-default-7884dbdc78-766jz"]
Sep 10 13:58:46.064135 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:46.064097    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" path="/var/lib/kubelet/pods/3c9f321f-c6be-4cd9-9454-b20c4436ce9f/volumes"
Sep 10 13:58:49.421161 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.421124    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:58:49.421691 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.421350    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" podUID="22e8d7c8-6241-4f33-9cc6-feed7996278b" containerName="maas-api" containerID="cri-o://a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae" gracePeriod=30
Sep 10 13:58:49.678997 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.678921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:58:49.776799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.776763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") pod \"22e8d7c8-6241-4f33-9cc6-feed7996278b\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") "
Sep 10 13:58:49.776799 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.776802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca\") pod \"22e8d7c8-6241-4f33-9cc6-feed7996278b\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") "
Sep 10 13:58:49.777034 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.776843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86vfv\" (UniqueName: \"kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv\") pod \"22e8d7c8-6241-4f33-9cc6-feed7996278b\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") "
Sep 10 13:58:49.777034 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.776946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs\") pod \"22e8d7c8-6241-4f33-9cc6-feed7996278b\" (UID: \"22e8d7c8-6241-4f33-9cc6-feed7996278b\") "
Sep 10 13:58:49.777212 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.777186    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22e8d7c8-6241-4f33-9cc6-feed7996278b" (UID: "22e8d7c8-6241-4f33-9cc6-feed7996278b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:58:49.777331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.777314    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22e8d7c8-6241-4f33-9cc6-feed7996278b-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:49.779049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.779019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22e8d7c8-6241-4f33-9cc6-feed7996278b" (UID: "22e8d7c8-6241-4f33-9cc6-feed7996278b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:58:49.779049 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.779035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv" (OuterVolumeSpecName: "kube-api-access-86vfv") pod "22e8d7c8-6241-4f33-9cc6-feed7996278b" (UID: "22e8d7c8-6241-4f33-9cc6-feed7996278b"). InnerVolumeSpecName "kube-api-access-86vfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:58:49.779179 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.779035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22e8d7c8-6241-4f33-9cc6-feed7996278b" (UID: "22e8d7c8-6241-4f33-9cc6-feed7996278b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:58:49.878657 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.878625    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:49.878657 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.878653    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22e8d7c8-6241-4f33-9cc6-feed7996278b-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:49.878657 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:49.878663    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86vfv\" (UniqueName: \"kubernetes.io/projected/22e8d7c8-6241-4f33-9cc6-feed7996278b-kube-api-access-86vfv\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:58:50.403829 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.403794    2577 generic.go:356] "Generic (PLEG): container finished" podID="22e8d7c8-6241-4f33-9cc6-feed7996278b" containerID="a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae" exitCode=0
Sep 10 13:58:50.404070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.403841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"
Sep 10 13:58:50.404070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.403886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" event={"ID":"22e8d7c8-6241-4f33-9cc6-feed7996278b","Type":"ContainerDied","Data":"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"}
Sep 10 13:58:50.404070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.403944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq" event={"ID":"22e8d7c8-6241-4f33-9cc6-feed7996278b","Type":"ContainerDied","Data":"1deb017767649d5bcf65aacbaf73563f2680ebf007188bb14573736bbf5ed584"}
Sep 10 13:58:50.404070 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.403972    2577 scope.go:117] "RemoveContainer" containerID="a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"
Sep 10 13:58:50.412820 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.412801    2577 scope.go:117] "RemoveContainer" containerID="a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"
Sep 10 13:58:50.413115 ip-10-0-133-48 kubenswrapper[2577]: E0910 13:58:50.413079    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae\": container with ID starting with a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae not found: ID does not exist" containerID="a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"
Sep 10 13:58:50.413191 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.413110    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae"} err="failed to get container status \"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae\": rpc error: code = NotFound desc = could not find container \"a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae\": container with ID starting with a32e2532676dc5ce5318bf121bc51800b8331be2e7598a2365534cfed61015ae not found: ID does not exist"
Sep 10 13:58:50.423581 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.423553    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:58:50.431098 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:50.431070    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a310dd-99b69587f-lrhvq"]
Sep 10 13:58:52.063920 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:52.063885    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e8d7c8-6241-4f33-9cc6-feed7996278b" path="/var/lib/kubelet/pods/22e8d7c8-6241-4f33-9cc6-feed7996278b/volumes"
Sep 10 13:58:54.237412 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:58:54.237372    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:59:03.944971 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:03.944918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:59:14.137633 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:14.137575    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:59:18.434531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:18.434470    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:59:19.422882 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:19.422844    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:59:23.258599 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:23.258559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:59:25.995095 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995055    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 13:59:25.995478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995455    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e8d7c8-6241-4f33-9cc6-feed7996278b" containerName="maas-api"
Sep 10 13:59:25.995478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995467    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e8d7c8-6241-4f33-9cc6-feed7996278b" containerName="maas-api"
Sep 10 13:59:25.995478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995475    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:59:25.995478 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995480    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995499    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995518    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995530    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995535    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995592    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e44e833-3bfb-4537-91e1-ac799630c530" containerName="istio-proxy"
Sep 10 13:59:25.995639 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.995610    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e8d7c8-6241-4f33-9cc6-feed7996278b" containerName="maas-api"
Sep 10 13:59:25.998410 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:25.998392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.001574 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.001554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3f01c7-dockercfg-kbdsg\""
Sep 10 13:59:26.001871 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.001855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3f01c7\""
Sep 10 13:59:26.011109 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.011080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 13:59:26.096952 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.096922    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 13:59:26.097397 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.097383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9f321f-c6be-4cd9-9454-b20c4436ce9f" containerName="istio-proxy"
Sep 10 13:59:26.100498 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.100478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.115923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.115899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 13:59:26.137748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.137707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.137923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.137762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8ks\" (UniqueName: \"kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.238855 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.238804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8ks\" (UniqueName: \"kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.239059 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.238932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmnr\" (UniqueName: \"kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.239059 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.238967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.239059 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.239019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.239751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.239724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.251737 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.251667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8ks\" (UniqueName: \"kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks\") pod \"payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.308732 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.308690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:26.340267 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.340229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmnr\" (UniqueName: \"kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.340427 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.340274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.340984 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.340963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.361040 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.360734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmnr\" (UniqueName: \"kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr\") pod \"payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.410762 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.410726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:26.585578 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.585554    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 13:59:26.586908 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:59:26.586880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a61137_9283_4646_a8c1_36176b99f799.slice/crio-f43446d439a6937f69a6f2b00c8f48008b5c2da57d6e9c807fae2c51002538a2 WatchSource:0}: Error finding container f43446d439a6937f69a6f2b00c8f48008b5c2da57d6e9c807fae2c51002538a2: Status 404 returned error can't find the container with id f43446d439a6937f69a6f2b00c8f48008b5c2da57d6e9c807fae2c51002538a2
Sep 10 13:59:26.705411 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:26.705385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 13:59:26.707783 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:59:26.707758    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6b4203_4c8f_4cfc_9f97_57df0ec57e6e.slice/crio-0e297c1f664cbbee7115afb282b4d1430f72d9efeb07d71275cfe170eee7d600 WatchSource:0}: Error finding container 0e297c1f664cbbee7115afb282b4d1430f72d9efeb07d71275cfe170eee7d600: Status 404 returned error can't find the container with id 0e297c1f664cbbee7115afb282b4d1430f72d9efeb07d71275cfe170eee7d600
Sep 10 13:59:27.591305 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.591259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" event={"ID":"e7a61137-9283-4646-a8c1-36176b99f799","Type":"ContainerStarted","Data":"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"}
Sep 10 13:59:27.591792 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.591335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" event={"ID":"e7a61137-9283-4646-a8c1-36176b99f799","Type":"ContainerStarted","Data":"f43446d439a6937f69a6f2b00c8f48008b5c2da57d6e9c807fae2c51002538a2"}
Sep 10 13:59:27.591792 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.591370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:27.592949 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.592920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" event={"ID":"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e","Type":"ContainerStarted","Data":"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"}
Sep 10 13:59:27.593071 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.592969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" event={"ID":"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e","Type":"ContainerStarted","Data":"0e297c1f664cbbee7115afb282b4d1430f72d9efeb07d71275cfe170eee7d600"}
Sep 10 13:59:27.593071 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.593014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:27.626025 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.625969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" podStartSLOduration=2.625953718 podStartE2EDuration="2.625953718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:27.624661368 +0000 UTC m=+1852.079536531" watchObservedRunningTime="2026-09-10 13:59:27.625953718 +0000 UTC m=+1852.080828881"
Sep 10 13:59:27.663945 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:27.663893    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" podStartSLOduration=1.663879075 podStartE2EDuration="1.663879075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:27.663296774 +0000 UTC m=+1852.118171938" watchObservedRunningTime="2026-09-10 13:59:27.663879075 +0000 UTC m=+1852.118754238"
Sep 10 13:59:33.454221 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.454173    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" podUID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" containerName="main" containerID="cri-o://aee9f7c653e45ce1d5a1269e5f86d366c77785fcc9a7043b5189ed7bff49e4ba" gracePeriod=45
Sep 10 13:59:33.626670 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.626640    2577 generic.go:356] "Generic (PLEG): container finished" podID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" containerID="aee9f7c653e45ce1d5a1269e5f86d366c77785fcc9a7043b5189ed7bff49e4ba" exitCode=0
Sep 10 13:59:33.626831 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.626692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" event={"ID":"2ec080e0-61b2-4847-b47b-6d09ad342d0a","Type":"ContainerDied","Data":"aee9f7c653e45ce1d5a1269e5f86d366c77785fcc9a7043b5189ed7bff49e4ba"}
Sep 10 13:59:33.703849 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.703817    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:59:33.813115 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813115 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dwst\" (UniqueName: \"kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813318 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs\") pod \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\" (UID: \"2ec080e0-61b2-4847-b47b-6d09ad342d0a\") "
Sep 10 13:59:33.813477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:33.813477 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813450    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home" (OuterVolumeSpecName: "home") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:33.813625 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:33.813697 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813676    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.813754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813698    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.813754 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.813706    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.815331 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.815298    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:59:33.815455 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.815351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm" (OuterVolumeSpecName: "dshm") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:33.815455 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.815364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst" (OuterVolumeSpecName: "kube-api-access-7dwst") pod "2ec080e0-61b2-4847-b47b-6d09ad342d0a" (UID: "2ec080e0-61b2-4847-b47b-6d09ad342d0a"). InnerVolumeSpecName "kube-api-access-7dwst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:59:33.914457 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.914422    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ec080e0-61b2-4847-b47b-6d09ad342d0a-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.914457 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.914452    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ec080e0-61b2-4847-b47b-6d09ad342d0a-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.914457 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.914463    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dwst\" (UniqueName: \"kubernetes.io/projected/2ec080e0-61b2-4847-b47b-6d09ad342d0a-kube-api-access-7dwst\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:33.990276 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:33.990242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 13:59:34.446089 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.446046    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" podUID="f7d01101-ac6f-4523-99f0-8fb3907627b7" containerName="main" containerID="cri-o://e1c2d8b3a50205455dbb6e2688deef4bc3bb8c776311a9bcc4692c6b91b603d9" gracePeriod=45
Sep 10 13:59:34.634131 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.634080    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"
Sep 10 13:59:34.634131 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.634077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g" event={"ID":"2ec080e0-61b2-4847-b47b-6d09ad342d0a","Type":"ContainerDied","Data":"85b10bb804649bb5d29581cdc4a8e510a6d5f453312fde6fa04788dbcc7a30ca"}
Sep 10 13:59:34.634738 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.634181    2577 scope.go:117] "RemoveContainer" containerID="aee9f7c653e45ce1d5a1269e5f86d366c77785fcc9a7043b5189ed7bff49e4ba"
Sep 10 13:59:34.636974 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.636737    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7d01101-ac6f-4523-99f0-8fb3907627b7" containerID="e1c2d8b3a50205455dbb6e2688deef4bc3bb8c776311a9bcc4692c6b91b603d9" exitCode=0
Sep 10 13:59:34.636974 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.636918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" event={"ID":"f7d01101-ac6f-4523-99f0-8fb3907627b7","Type":"ContainerDied","Data":"e1c2d8b3a50205455dbb6e2688deef4bc3bb8c776311a9bcc4692c6b91b603d9"}
Sep 10 13:59:34.654213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.654184    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:59:34.659659 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.659631    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7a11e9/rate-test-model-7a11e9-kserve-5b84c86cf-pm82g"]
Sep 10 13:59:34.693551 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.693526    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:59:34.824013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.823914    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2tjw\" (UniqueName: \"kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824013 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.823992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824213 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir\") pod \"f7d01101-ac6f-4523-99f0-8fb3907627b7\" (UID: \"f7d01101-ac6f-4523-99f0-8fb3907627b7\") "
Sep 10 13:59:34.824418 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:34.824480 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824383    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home" (OuterVolumeSpecName: "home") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:34.824569 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:34.824783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.824759    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:34.826185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.826155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:59:34.826185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.826166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm" (OuterVolumeSpecName: "dshm") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:59:34.826322 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.826239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw" (OuterVolumeSpecName: "kube-api-access-j2tjw") pod "f7d01101-ac6f-4523-99f0-8fb3907627b7" (UID: "f7d01101-ac6f-4523-99f0-8fb3907627b7"). InnerVolumeSpecName "kube-api-access-j2tjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:59:34.926185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.926147    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:34.926185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.926179    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:34.926185 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.926188    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7d01101-ac6f-4523-99f0-8fb3907627b7-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:34.926430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.926196    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2tjw\" (UniqueName: \"kubernetes.io/projected/f7d01101-ac6f-4523-99f0-8fb3907627b7-kube-api-access-j2tjw\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:34.926430 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:34.926206    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7d01101-ac6f-4523-99f0-8fb3907627b7-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 13:59:35.643983 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:35.643952    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"
Sep 10 13:59:35.643983 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:35.643969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c" event={"ID":"f7d01101-ac6f-4523-99f0-8fb3907627b7","Type":"ContainerDied","Data":"0be06c133fb420af51cd0cfdbf4c5aea990835c6b62e93585231b281be3005e0"}
Sep 10 13:59:35.644471 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:35.644021    2577 scope.go:117] "RemoveContainer" containerID="e1c2d8b3a50205455dbb6e2688deef4bc3bb8c776311a9bcc4692c6b91b603d9"
Sep 10 13:59:35.666531 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:35.666480    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:59:35.673570 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:35.673543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a00003/rate-test-model-a00003-kserve-7856fc8dc-hnk6c"]
Sep 10 13:59:36.065258 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:36.065178    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" path="/var/lib/kubelet/pods/2ec080e0-61b2-4847-b47b-6d09ad342d0a/volumes"
Sep 10 13:59:36.065577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:36.065563    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d01101-ac6f-4523-99f0-8fb3907627b7" path="/var/lib/kubelet/pods/f7d01101-ac6f-4523-99f0-8fb3907627b7/volumes"
Sep 10 13:59:38.601248 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:38.601214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 13:59:38.601715 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:38.601542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 13:59:49.858197 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858162    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858602    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" containerName="main"
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858616    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" containerName="main"
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858634    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d01101-ac6f-4523-99f0-8fb3907627b7" containerName="main"
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858640    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d01101-ac6f-4523-99f0-8fb3907627b7" containerName="main"
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858699    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec080e0-61b2-4847-b47b-6d09ad342d0a" containerName="main"
Sep 10 13:59:49.858743 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.858707    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d01101-ac6f-4523-99f0-8fb3907627b7" containerName="main"
Sep 10 13:59:49.861767 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.861656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.865052 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.865025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b0741a-openshift-default-dockercfg-nqvvp\""
Sep 10 13:59:49.879751 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.879720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 13:59:49.960796 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpcl\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.960982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.960982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.960982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960912    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.960982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.960982 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.961156 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.960998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.961156 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.961078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:49.961156 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:49.961116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.061923 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.061886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.061940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.061959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpcl\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062129 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062433 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062433 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062637 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062637 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.062915 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.063028 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.062970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.064596 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.064576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.064907 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.064890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.075188 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.075156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.075289 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.075204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpcl\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl\") pod \"e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") " pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.171609 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.171570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:50.449941 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.449912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 13:59:50.452194 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:59:50.452159    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2055234_c5c4_472b_95ac_5147dac379bc.slice/crio-776aadd392b42d5e581b3453caceb3eafd5842b4d312c435e57074c8c44d0d70 WatchSource:0}: Error finding container 776aadd392b42d5e581b3453caceb3eafd5842b4d312c435e57074c8c44d0d70: Status 404 returned error can't find the container with id 776aadd392b42d5e581b3453caceb3eafd5842b4d312c435e57074c8c44d0d70
Sep 10 13:59:50.454469 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.454431    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:59:50.454576 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.454518    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:59:50.454576 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.454556    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 13:59:50.720067 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.719983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerStarted","Data":"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"}
Sep 10 13:59:50.720067 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.720033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerStarted","Data":"776aadd392b42d5e581b3453caceb3eafd5842b4d312c435e57074c8c44d0d70"}
Sep 10 13:59:50.745748 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:50.745683    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podStartSLOduration=1.745660185 podStartE2EDuration="1.745660185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:50.743092408 +0000 UTC m=+1875.197967592" watchObservedRunningTime="2026-09-10 13:59:50.745660185 +0000 UTC m=+1875.200535349"
Sep 10 13:59:51.172726 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:51.172693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:51.177899 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:51.177874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:51.725577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:51.725537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:51.726432 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:51.726408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 13:59:55.249742 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.249709    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 13:59:55.253498 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.253477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.255934 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.255910    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b0741a-dockercfg-tdpcv\""
Sep 10 13:59:55.256266 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.256239    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b0741a\""
Sep 10 13:59:55.263421 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.263392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 13:59:55.310577 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.310547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqk4\" (UniqueName: \"kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.310764 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.310592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.412037 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.411996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqk4\" (UniqueName: \"kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.412250 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.412050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.412794 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.412773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.423814 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.423781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqk4\" (UniqueName: \"kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4\") pod \"payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.565237 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.565141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:55.856095 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:55.856066    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 13:59:55.858397 ip-10-0-133-48 kubenswrapper[2577]: W0910 13:59:55.858371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda858bcf_76c2_4619_b149_7bd48f224a51.slice/crio-4d61e5ee96a1c7a81426b563129e71af0825864b15713a22cac524ea82193f7f WatchSource:0}: Error finding container 4d61e5ee96a1c7a81426b563129e71af0825864b15713a22cac524ea82193f7f: Status 404 returned error can't find the container with id 4d61e5ee96a1c7a81426b563129e71af0825864b15713a22cac524ea82193f7f
Sep 10 13:59:56.754005 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:56.753964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" event={"ID":"da858bcf-76c2-4619-b149-7bd48f224a51","Type":"ContainerStarted","Data":"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"}
Sep 10 13:59:56.754437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:56.754022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" event={"ID":"da858bcf-76c2-4619-b149-7bd48f224a51","Type":"ContainerStarted","Data":"4d61e5ee96a1c7a81426b563129e71af0825864b15713a22cac524ea82193f7f"}
Sep 10 13:59:56.754437 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:56.754098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 13:59:56.773033 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:56.772933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" podStartSLOduration=1.7729092579999999 podStartE2EDuration="1.772909258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:59:56.772361539 +0000 UTC m=+1881.227236700" watchObservedRunningTime="2026-09-10 13:59:56.772909258 +0000 UTC m=+1881.227784420"
Sep 10 13:59:59.439783 ip-10-0-133-48 kubenswrapper[2577]: I0910 13:59:59.439698    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:00:07.762279 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:07.762240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 14:00:07.766709 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:07.766668    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 14:00:07.766869 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:07.766745    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:00:09.822665 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.822630    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2055234-c5c4-472b-95ac-5147dac379bc" containerID="b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10" exitCode=0
Sep 10 14:00:09.823096 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.822707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerDied","Data":"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"}
Sep 10 14:00:09.823140 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.823129    2577 scope.go:117] "RemoveContainer" containerID="b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"
Sep 10 14:00:09.824291 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.824244    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 14:00:09.824359 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.824334    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 14:00:09.824397 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:09.824364    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 14:00:10.829108 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:10.829065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerStarted","Data":"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"}
Sep 10 14:00:11.171782 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:11.171741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:00:11.172035 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:11.172010    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 14:00:11.172091 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:11.172065    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 14:00:12.172926 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:12.172884    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 14:00:12.173338 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:12.172946    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 14:00:13.172006 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:13.171967    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 14:00:13.172203 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:13.172025    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 14:00:14.172210 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:14.172170    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 14:00:14.172650 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:14.172229    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 14:00:15.172815 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:15.172768    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 10 14:00:15.173314 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:15.172837    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 10 14:00:16.176533 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:16.176474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:00:16.176985 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:16.176778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:00:16.177570 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:16.177551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:00:27.796167 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.796128    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:00:27.800299 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.800276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.802886 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.802862    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3f01c7\"/\"openshift-service-ca.crt\""
Sep 10 14:00:27.803715 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.803698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3f01c7\"/\"kube-root-ca.crt\""
Sep 10 14:00:27.803796 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.803755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3f01c7\"/\"ipp-route-3f01c7-kserve-self-signed-certs\""
Sep 10 14:00:27.803837 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.803801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3f01c7\"/\"default-dockercfg-l8p46\""
Sep 10 14:00:27.814187 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.814158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:00:27.936245 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.936449 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.936449 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.936449 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.936449 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmlm\" (UniqueName: \"kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:27.936696 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:27.936551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038004 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.037960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmlm\" (UniqueName: \"kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038208 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038208 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038208 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038208 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038422 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038602 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038698 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.038698 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.038686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.040847 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.040818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.040956 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.040897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.055715 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.055654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmlm\" (UniqueName: \"kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm\") pod \"ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") " pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.109684 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.109646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:28.386905 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.386876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:00:28.388797 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:00:28.388766    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod080f8b9b_c6bf_449d_848c_5719e9c059e3.slice/crio-2afbc9d87501655819d313c4bba674313661201070794aed6080c77c7192461c WatchSource:0}: Error finding container 2afbc9d87501655819d313c4bba674313661201070794aed6080c77c7192461c: Status 404 returned error can't find the container with id 2afbc9d87501655819d313c4bba674313661201070794aed6080c77c7192461c
Sep 10 14:00:28.932001 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.931961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" event={"ID":"080f8b9b-c6bf-449d-848c-5719e9c059e3","Type":"ContainerStarted","Data":"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"}
Sep 10 14:00:28.932500 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.932015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" event={"ID":"080f8b9b-c6bf-449d-848c-5719e9c059e3","Type":"ContainerStarted","Data":"2afbc9d87501655819d313c4bba674313661201070794aed6080c77c7192461c"}
Sep 10 14:00:28.961831 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:28.961773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" podStartSLOduration=1.961757617 podStartE2EDuration="1.961757617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:00:28.959179879 +0000 UTC m=+1913.414055041" watchObservedRunningTime="2026-09-10 14:00:28.961757617 +0000 UTC m=+1913.416632780"
Sep 10 14:00:31.855310 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:31.855268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:00:38.110150 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:38.110110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:38.110666 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:38.110385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:38.123090 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:38.123058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:38.998856 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:38.998829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:00:47.358605 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:47.358560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:00:53.591401 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:53.591354    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq_ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e/payload-processing/0.log"
Sep 10 14:00:59.222495 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.222456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:00:59.226292 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.226272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.228611 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.228587    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eb3d6d\""
Sep 10 14:00:59.236352 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.236323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:00:59.351320 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.351282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmkz\" (UniqueName: \"kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.351533 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.351338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.452555 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.452490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmkz\" (UniqueName: \"kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.452705 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.452584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.453282 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.453259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.462278 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.462247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmkz\" (UniqueName: \"kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz\") pod \"payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.538852 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.538753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:00:59.814563 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:00:59.814531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:00:59.817320 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:00:59.817290    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcf1dab_ef1d_4baf_8b71_b031c4e3474d.slice/crio-9fe004b900952b071fa9301205336bbe7f9a02e7bf2809ecbe9667c2021c91e7 WatchSource:0}: Error finding container 9fe004b900952b071fa9301205336bbe7f9a02e7bf2809ecbe9667c2021c91e7: Status 404 returned error can't find the container with id 9fe004b900952b071fa9301205336bbe7f9a02e7bf2809ecbe9667c2021c91e7
Sep 10 14:01:00.095883 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:00.095790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" event={"ID":"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d","Type":"ContainerStarted","Data":"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"}
Sep 10 14:01:00.095883 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:00.095852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" event={"ID":"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d","Type":"ContainerStarted","Data":"9fe004b900952b071fa9301205336bbe7f9a02e7bf2809ecbe9667c2021c91e7"}
Sep 10 14:01:00.096103 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:00.095945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:01:00.116318 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:00.116266    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" podStartSLOduration=1.116249631 podStartE2EDuration="1.116249631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:01:00.113906683 +0000 UTC m=+1944.568781845" watchObservedRunningTime="2026-09-10 14:01:00.116249631 +0000 UTC m=+1944.571124820"
Sep 10 14:01:07.049435 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:07.049394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:01:11.103174 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:11.103134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:01:29.553287 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.553202    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:01:29.553764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.553467    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" podUID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" containerName="payload-pre-processing" containerID="cri-o://dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985" gracePeriod=30
Sep 10 14:01:29.810290 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.810227    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:01:29.943801 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.943767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume\") pod \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") "
Sep 10 14:01:29.943969 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.943888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkmkz\" (UniqueName: \"kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz\") pod \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\" (UID: \"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d\") "
Sep 10 14:01:29.944229 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.944203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" (UID: "8fcf1dab-ef1d-4baf-8b71-b031c4e3474d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:01:29.945928 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:29.945908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz" (OuterVolumeSpecName: "kube-api-access-jkmkz") pod "8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" (UID: "8fcf1dab-ef1d-4baf-8b71-b031c4e3474d"). InnerVolumeSpecName "kube-api-access-jkmkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:01:30.045382 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.045349    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:30.045382 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.045377    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmkz\" (UniqueName: \"kubernetes.io/projected/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d-kube-api-access-jkmkz\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:30.254757 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.254722    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" containerID="dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985" exitCode=0
Sep 10 14:01:30.254936 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.254768    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"
Sep 10 14:01:30.254936 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.254812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" event={"ID":"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d","Type":"ContainerDied","Data":"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"}
Sep 10 14:01:30.254936 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.254864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t" event={"ID":"8fcf1dab-ef1d-4baf-8b71-b031c4e3474d","Type":"ContainerDied","Data":"9fe004b900952b071fa9301205336bbe7f9a02e7bf2809ecbe9667c2021c91e7"}
Sep 10 14:01:30.254936 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.254886    2577 scope.go:117] "RemoveContainer" containerID="dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"
Sep 10 14:01:30.263399 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.263381    2577 scope.go:117] "RemoveContainer" containerID="dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"
Sep 10 14:01:30.263769 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:01:30.263752    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985\": container with ID starting with dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985 not found: ID does not exist" containerID="dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"
Sep 10 14:01:30.263817 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.263780    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985"} err="failed to get container status \"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985\": rpc error: code = NotFound desc = could not find container \"dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985\": container with ID starting with dd1153c1ccb76025ab2dd9bdf1659ddac6f04079893670ee080d4ee018b7e985 not found: ID does not exist"
Sep 10 14:01:30.298385 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.298354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:01:30.303589 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:30.303565    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eb3d6d-56bc754658-4s94t"]
Sep 10 14:01:32.064564 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:32.064525    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" path="/var/lib/kubelet/pods/8fcf1dab-ef1d-4baf-8b71-b031c4e3474d/volumes"
Sep 10 14:01:34.054445 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:34.054409    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:01:55.387355 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:55.387319    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:01:57.918406 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:57.918367    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 14:01:57.918924 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:57.918649    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" podUID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" containerName="payload-processing" containerID="cri-o://ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6" gracePeriod=30
Sep 10 14:01:57.938910 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:57.938273    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 14:01:57.938910 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:57.938532    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" podUID="e7a61137-9283-4646-a8c1-36176b99f799" containerName="payload-pre-processing" containerID="cri-o://df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359" gracePeriod=30
Sep 10 14:01:58.209533 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.209498    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 14:01:58.212104 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.212085    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 14:01:58.317038 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.316992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume\") pod \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") "
Sep 10 14:01:58.317038 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.317041    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmnr\" (UniqueName: \"kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr\") pod \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\" (UID: \"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e\") "
Sep 10 14:01:58.317293 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.317092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume\") pod \"e7a61137-9283-4646-a8c1-36176b99f799\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") "
Sep 10 14:01:58.317293 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.317110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8ks\" (UniqueName: \"kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks\") pod \"e7a61137-9283-4646-a8c1-36176b99f799\" (UID: \"e7a61137-9283-4646-a8c1-36176b99f799\") "
Sep 10 14:01:58.317493 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.317465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" (UID: "ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:01:58.317589 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.317464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7a61137-9283-4646-a8c1-36176b99f799" (UID: "e7a61137-9283-4646-a8c1-36176b99f799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:01:58.319359 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.319329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr" (OuterVolumeSpecName: "kube-api-access-bqmnr") pod "ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" (UID: "ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e"). InnerVolumeSpecName "kube-api-access-bqmnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:01:58.319359 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.319338    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks" (OuterVolumeSpecName: "kube-api-access-lz8ks") pod "e7a61137-9283-4646-a8c1-36176b99f799" (UID: "e7a61137-9283-4646-a8c1-36176b99f799"). InnerVolumeSpecName "kube-api-access-lz8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:01:58.396863 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.396816    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7a61137-9283-4646-a8c1-36176b99f799" containerID="df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359" exitCode=0
Sep 10 14:01:58.397040 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.396886    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"
Sep 10 14:01:58.397040 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.396906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" event={"ID":"e7a61137-9283-4646-a8c1-36176b99f799","Type":"ContainerDied","Data":"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"}
Sep 10 14:01:58.397040 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.396989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck" event={"ID":"e7a61137-9283-4646-a8c1-36176b99f799","Type":"ContainerDied","Data":"f43446d439a6937f69a6f2b00c8f48008b5c2da57d6e9c807fae2c51002538a2"}
Sep 10 14:01:58.397040 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.397018    2577 scope.go:117] "RemoveContainer" containerID="df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"
Sep 10 14:01:58.398406 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.398384    2577 generic.go:356] "Generic (PLEG): container finished" podID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" containerID="ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6" exitCode=0
Sep 10 14:01:58.398490 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.398441    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"
Sep 10 14:01:58.398490 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.398460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" event={"ID":"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e","Type":"ContainerDied","Data":"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"}
Sep 10 14:01:58.398586 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.398532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq" event={"ID":"ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e","Type":"ContainerDied","Data":"0e297c1f664cbbee7115afb282b4d1430f72d9efeb07d71275cfe170eee7d600"}
Sep 10 14:01:58.408201 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.408175    2577 scope.go:117] "RemoveContainer" containerID="df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"
Sep 10 14:01:58.408961 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:01:58.408935    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359\": container with ID starting with df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359 not found: ID does not exist" containerID="df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"
Sep 10 14:01:58.409087 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.408966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359"} err="failed to get container status \"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359\": rpc error: code = NotFound desc = could not find container \"df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359\": container with ID starting with df44c15d72ad31dce294257daa9ae6ea178c93c37cc09c58f3d4308a34265359 not found: ID does not exist"
Sep 10 14:01:58.409087 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.408988    2577 scope.go:117] "RemoveContainer" containerID="ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"
Sep 10 14:01:58.417226 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417202    2577 scope.go:117] "RemoveContainer" containerID="ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"
Sep 10 14:01:58.417519 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:01:58.417489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6\": container with ID starting with ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6 not found: ID does not exist" containerID="ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"
Sep 10 14:01:58.417615 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417538    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6"} err="failed to get container status \"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6\": rpc error: code = NotFound desc = could not find container \"ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6\": container with ID starting with ba5eded1fe36bbb537063d9a9752b3b8673043f2c8f9853aeef6d5d120a882d6 not found: ID does not exist"
Sep 10 14:01:58.417969 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417955    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a61137-9283-4646-a8c1-36176b99f799-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:58.418011 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417972    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8ks\" (UniqueName: \"kubernetes.io/projected/e7a61137-9283-4646-a8c1-36176b99f799-kube-api-access-lz8ks\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:58.418011 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417981    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:58.418011 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.417989    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmnr\" (UniqueName: \"kubernetes.io/projected/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e-kube-api-access-bqmnr\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:01:58.436782 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.436711    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 14:01:58.449245 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.449218    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3f01c7-9bd6d7bcd-db8ck"]
Sep 10 14:01:58.473269 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.473239    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 14:01:58.489289 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:01:58.489260    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3f01c7-69c8b8bd9c-dg5rq"]
Sep 10 14:02:00.063812 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:00.063771    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a61137-9283-4646-a8c1-36176b99f799" path="/var/lib/kubelet/pods/e7a61137-9283-4646-a8c1-36176b99f799/volumes"
Sep 10 14:02:00.064259 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:00.064243    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" path="/var/lib/kubelet/pods/ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e/volumes"
Sep 10 14:02:03.589854 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:03.589815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:02:09.104126 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:09.104087    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:02:18.609840 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:18.609782    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" podUID="080f8b9b-c6bf-449d-848c-5719e9c059e3" containerName="main" containerID="cri-o://d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0" gracePeriod=45
Sep 10 14:02:18.864446 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:18.864421    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:02:19.019406 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019586 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019586 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019674 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019630    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019719 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019775 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mmlm\" (UniqueName: \"kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm\") pod \"080f8b9b-c6bf-449d-848c-5719e9c059e3\" (UID: \"080f8b9b-c6bf-449d-848c-5719e9c059e3\") "
Sep 10 14:02:19.019775 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:19.019953 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home" (OuterVolumeSpecName: "home") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:19.020036 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.019951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:19.020129 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.020113    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.020205 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.020135    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.020205 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.020175    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.021751 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.021727    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:02:19.021751 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.021739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm" (OuterVolumeSpecName: "dshm") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:02:19.021892 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.021808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm" (OuterVolumeSpecName: "kube-api-access-7mmlm") pod "080f8b9b-c6bf-449d-848c-5719e9c059e3" (UID: "080f8b9b-c6bf-449d-848c-5719e9c059e3"). InnerVolumeSpecName "kube-api-access-7mmlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:02:19.121290 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.121203    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mmlm\" (UniqueName: \"kubernetes.io/projected/080f8b9b-c6bf-449d-848c-5719e9c059e3-kube-api-access-7mmlm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.121290 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.121236    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/080f8b9b-c6bf-449d-848c-5719e9c059e3-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.121290 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.121247    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/080f8b9b-c6bf-449d-848c-5719e9c059e3-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:02:19.511782 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.511685    2577 generic.go:356] "Generic (PLEG): container finished" podID="080f8b9b-c6bf-449d-848c-5719e9c059e3" containerID="d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0" exitCode=0
Sep 10 14:02:19.511782 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.511734    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"
Sep 10 14:02:19.511782 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.511765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" event={"ID":"080f8b9b-c6bf-449d-848c-5719e9c059e3","Type":"ContainerDied","Data":"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"}
Sep 10 14:02:19.512062 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.511808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v" event={"ID":"080f8b9b-c6bf-449d-848c-5719e9c059e3","Type":"ContainerDied","Data":"2afbc9d87501655819d313c4bba674313661201070794aed6080c77c7192461c"}
Sep 10 14:02:19.512062 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.511825    2577 scope.go:117] "RemoveContainer" containerID="d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"
Sep 10 14:02:19.520835 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.520818    2577 scope.go:117] "RemoveContainer" containerID="d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"
Sep 10 14:02:19.521090 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:02:19.521072    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0\": container with ID starting with d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0 not found: ID does not exist" containerID="d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"
Sep 10 14:02:19.521163 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.521097    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0"} err="failed to get container status \"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0\": rpc error: code = NotFound desc = could not find container \"d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0\": container with ID starting with d41f9ebeda5ce7b2513989973dd00ee3bf8a66a6bd721bc3dfbfd96da85685b0 not found: ID does not exist"
Sep 10 14:02:19.535672 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.535645    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:02:19.542001 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:19.541972    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3f01c7/ipp-route-3f01c7-kserve-cbcc66f6c-fdk4v"]
Sep 10 14:02:20.064337 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:20.064301    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="080f8b9b-c6bf-449d-848c-5719e9c059e3" path="/var/lib/kubelet/pods/080f8b9b-c6bf-449d-848c-5719e9c059e3/volumes"
Sep 10 14:02:36.369425 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:02:36.369390    2577 scope.go:117] "RemoveContainer" containerID="947f1a12591b5f3bf54cebf73cde0736ede9a357cf8d75bbb810784c11b86ce6"
Sep 10 14:03:01.329011 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.328971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 14:03:01.329457 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.329214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" podUID="da858bcf-76c2-4619-b149-7bd48f224a51" containerName="payload-pre-processing" containerID="cri-o://e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0" gracePeriod=30
Sep 10 14:03:01.582202 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.582177    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 14:03:01.720750 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.720712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msqk4\" (UniqueName: \"kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4\") pod \"da858bcf-76c2-4619-b149-7bd48f224a51\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") "
Sep 10 14:03:01.720968 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.720863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume\") pod \"da858bcf-76c2-4619-b149-7bd48f224a51\" (UID: \"da858bcf-76c2-4619-b149-7bd48f224a51\") "
Sep 10 14:03:01.721298 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.721268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da858bcf-76c2-4619-b149-7bd48f224a51" (UID: "da858bcf-76c2-4619-b149-7bd48f224a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:01.722042 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.722018    2577 generic.go:356] "Generic (PLEG): container finished" podID="da858bcf-76c2-4619-b149-7bd48f224a51" containerID="e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0" exitCode=0
Sep 10 14:03:01.722162 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.722064    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"
Sep 10 14:03:01.722162 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.722104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" event={"ID":"da858bcf-76c2-4619-b149-7bd48f224a51","Type":"ContainerDied","Data":"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"}
Sep 10 14:03:01.722280 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.722166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7" event={"ID":"da858bcf-76c2-4619-b149-7bd48f224a51","Type":"ContainerDied","Data":"4d61e5ee96a1c7a81426b563129e71af0825864b15713a22cac524ea82193f7f"}
Sep 10 14:03:01.722280 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.722201    2577 scope.go:117] "RemoveContainer" containerID="e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"
Sep 10 14:03:01.723076 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.723051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4" (OuterVolumeSpecName: "kube-api-access-msqk4") pod "da858bcf-76c2-4619-b149-7bd48f224a51" (UID: "da858bcf-76c2-4619-b149-7bd48f224a51"). InnerVolumeSpecName "kube-api-access-msqk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:01.737127 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.737102    2577 scope.go:117] "RemoveContainer" containerID="e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"
Sep 10 14:03:01.737559 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:03:01.737532    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0\": container with ID starting with e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0 not found: ID does not exist" containerID="e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"
Sep 10 14:03:01.737690 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.737574    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0"} err="failed to get container status \"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0\": rpc error: code = NotFound desc = could not find container \"e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0\": container with ID starting with e9bd8af2616f9f94ed5e62a104ea3118f0f61abee181c8ad12faae6a23083ee0 not found: ID does not exist"
Sep 10 14:03:01.821612 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.821525    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqk4\" (UniqueName: \"kubernetes.io/projected/da858bcf-76c2-4619-b149-7bd48f224a51-kube-api-access-msqk4\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:01.821612 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:01.821555    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da858bcf-76c2-4619-b149-7bd48f224a51-plugins-config-volume\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:02.051771 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:02.051736    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 14:03:02.065710 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:02.065679    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b0741a-7b89bf4ff4-c6vn7"]
Sep 10 14:03:04.064061 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:04.064028    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da858bcf-76c2-4619-b149-7bd48f224a51" path="/var/lib/kubelet/pods/da858bcf-76c2-4619-b149-7bd48f224a51/volumes"
Sep 10 14:03:04.473612 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:04.473578    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:03:10.688386 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:10.688344    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 14:03:10.689076 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:10.688641    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy" containerID="cri-o://92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358" gracePeriod=30
Sep 10 14:03:15.941338 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.941309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:03:15.957325 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957446 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957446 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957446 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bpcl\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957446 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:15.957703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket\") pod \"e2055234-c5c4-472b-95ac-5147dac379bc\" (UID: \"e2055234-c5c4-472b-95ac-5147dac379bc\") "
Sep 10 14:03:15.957960 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:15.958018 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.957996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:15.958073 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.958016    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:15.958073 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.958037    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2055234-c5c4-472b-95ac-5147dac379bc-istiod-ca-cert\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:15.958191 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.958164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:15.958267 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.958247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:15.959784 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.959757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:03:15.959901 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.959808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:03:15.960077 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.960058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl" (OuterVolumeSpecName: "kube-api-access-9bpcl") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "kube-api-access-9bpcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:15.960121 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:15.960081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2055234-c5c4-472b-95ac-5147dac379bc" (UID: "e2055234-c5c4-472b-95ac-5147dac379bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:16.059175 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059134    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2055234-c5c4-472b-95ac-5147dac379bc-istio-podinfo\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059175 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059163    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-workload-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059175 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059171    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-istio-token\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059175 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059180    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpcl\" (UniqueName: \"kubernetes.io/projected/e2055234-c5c4-472b-95ac-5147dac379bc-kube-api-access-9bpcl\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059549 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059190    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-data\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059549 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059198    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-istio-envoy\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.059549 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.059206    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2055234-c5c4-472b-95ac-5147dac379bc-credential-socket\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:16.808553 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.808492    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2055234-c5c4-472b-95ac-5147dac379bc" containerID="92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358" exitCode=0
Sep 10 14:03:16.808553 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.808545    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerDied","Data":"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"}
Sep 10 14:03:16.808789 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.808562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"
Sep 10 14:03:16.808789 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.808603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t" event={"ID":"e2055234-c5c4-472b-95ac-5147dac379bc","Type":"ContainerDied","Data":"776aadd392b42d5e581b3453caceb3eafd5842b4d312c435e57074c8c44d0d70"}
Sep 10 14:03:16.808789 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.808632    2577 scope.go:117] "RemoveContainer" containerID="92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"
Sep 10 14:03:16.817039 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.817020    2577 scope.go:117] "RemoveContainer" containerID="b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"
Sep 10 14:03:16.824637 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.824621    2577 scope.go:117] "RemoveContainer" containerID="92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"
Sep 10 14:03:16.824874 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:03:16.824844    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358\": container with ID starting with 92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358 not found: ID does not exist" containerID="92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"
Sep 10 14:03:16.824919 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.824888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358"} err="failed to get container status \"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358\": rpc error: code = NotFound desc = could not find container \"92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358\": container with ID starting with 92fa9123d1a7f2fb88393add694bb3894f7d6e4872f23d0e354104f8ee09c358 not found: ID does not exist"
Sep 10 14:03:16.824919 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.824909    2577 scope.go:117] "RemoveContainer" containerID="b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"
Sep 10 14:03:16.825157 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:03:16.825136    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10\": container with ID starting with b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10 not found: ID does not exist" containerID="b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"
Sep 10 14:03:16.825222 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.825166    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10"} err="failed to get container status \"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10\": rpc error: code = NotFound desc = could not find container \"b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10\": container with ID starting with b537464fa979040fa15e6cfba202a516b650b918e0b0840492b6ec9ca085ea10 not found: ID does not exist"
Sep 10 14:03:16.847752 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.847723    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 14:03:16.871167 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:16.871135    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b0741a-openshift-default-777bf97b98-w2h8t"]
Sep 10 14:03:18.069188 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:18.069153    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" path="/var/lib/kubelet/pods/e2055234-c5c4-472b-95ac-5147dac379bc/volumes"
Sep 10 14:03:22.165114 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.165077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 14:03:22.165621 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.165339    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" podUID="f914eea1-c419-4c92-ba25-edf4a5505a93" containerName="maas-api" containerID="cri-o://6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272" gracePeriod=30
Sep 10 14:03:22.417457 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.417395    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 14:03:22.520605 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.520571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") pod \"f914eea1-c419-4c92-ba25-edf4a5505a93\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") "
Sep 10 14:03:22.520803 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.520637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca\") pod \"f914eea1-c419-4c92-ba25-edf4a5505a93\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") "
Sep 10 14:03:22.520803 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.520792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs\") pod \"f914eea1-c419-4c92-ba25-edf4a5505a93\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") "
Sep 10 14:03:22.520953 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.520833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n64lj\" (UniqueName: \"kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj\") pod \"f914eea1-c419-4c92-ba25-edf4a5505a93\" (UID: \"f914eea1-c419-4c92-ba25-edf4a5505a93\") "
Sep 10 14:03:22.521011 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.520945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f914eea1-c419-4c92-ba25-edf4a5505a93" (UID: "f914eea1-c419-4c92-ba25-edf4a5505a93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:22.521186 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.521161    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f914eea1-c419-4c92-ba25-edf4a5505a93-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.522868 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.522845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f914eea1-c419-4c92-ba25-edf4a5505a93" (UID: "f914eea1-c419-4c92-ba25-edf4a5505a93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:22.522954 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.522873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f914eea1-c419-4c92-ba25-edf4a5505a93" (UID: "f914eea1-c419-4c92-ba25-edf4a5505a93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:22.522954 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.522875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj" (OuterVolumeSpecName: "kube-api-access-n64lj") pod "f914eea1-c419-4c92-ba25-edf4a5505a93" (UID: "f914eea1-c419-4c92-ba25-edf4a5505a93"). InnerVolumeSpecName "kube-api-access-n64lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:22.622251 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.622215    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.622251 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.622245    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f914eea1-c419-4c92-ba25-edf4a5505a93-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.622251 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.622255    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64lj\" (UniqueName: \"kubernetes.io/projected/f914eea1-c419-4c92-ba25-edf4a5505a93-kube-api-access-n64lj\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:22.842428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.842299    2577 generic.go:356] "Generic (PLEG): container finished" podID="f914eea1-c419-4c92-ba25-edf4a5505a93" containerID="6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272" exitCode=0
Sep 10 14:03:22.842428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.842362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" event={"ID":"f914eea1-c419-4c92-ba25-edf4a5505a93","Type":"ContainerDied","Data":"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"}
Sep 10 14:03:22.842428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.842366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"
Sep 10 14:03:22.842428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.842407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8" event={"ID":"f914eea1-c419-4c92-ba25-edf4a5505a93","Type":"ContainerDied","Data":"8e4901b13f9d5ae2f940a64efc6ba6ff2773cf713cbfba9dbfd270d891bfac84"}
Sep 10 14:03:22.842428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.842428    2577 scope.go:117] "RemoveContainer" containerID="6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"
Sep 10 14:03:22.851966 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.851950    2577 scope.go:117] "RemoveContainer" containerID="6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"
Sep 10 14:03:22.852223 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:03:22.852202    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272\": container with ID starting with 6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272 not found: ID does not exist" containerID="6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"
Sep 10 14:03:22.852293 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.852228    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272"} err="failed to get container status \"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272\": rpc error: code = NotFound desc = could not find container \"6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272\": container with ID starting with 6270cb9292a3b517cd32faa2ca68c02db846ae7a7d2600121fd5ea2205fa1272 not found: ID does not exist"
Sep 10 14:03:22.870760 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.870715    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 14:03:22.877845 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:22.877824    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a11e9-7759d4d68-hqrr8"]
Sep 10 14:03:24.063462 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:24.063428    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f914eea1-c419-4c92-ba25-edf4a5505a93" path="/var/lib/kubelet/pods/f914eea1-c419-4c92-ba25-edf4a5505a93/volumes"
Sep 10 14:03:24.321967 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:24.321874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:03:36.170334 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:36.170308    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 14:03:36.174937 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:36.174916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 14:03:41.782526 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:41.782472    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 14:03:41.783154 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:41.782797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" podUID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" containerName="maas-api" containerID="cri-o://b19dd7c22bbe2462e7dc03d65dbdc8b11c8b807fe06e5e78433cb0521f75301a" gracePeriod=30
Sep 10 14:03:41.934712 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:41.934679    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" containerID="b19dd7c22bbe2462e7dc03d65dbdc8b11c8b807fe06e5e78433cb0521f75301a" exitCode=0
Sep 10 14:03:41.934912 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:41.934743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" event={"ID":"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a","Type":"ContainerDied","Data":"b19dd7c22bbe2462e7dc03d65dbdc8b11c8b807fe06e5e78433cb0521f75301a"}
Sep 10 14:03:42.045906 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.045839    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 14:03:42.205357 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.205315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca\") pod \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") "
Sep 10 14:03:42.205577 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.205393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv74m\" (UniqueName: \"kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m\") pod \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") "
Sep 10 14:03:42.205577 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.205438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") pod \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") "
Sep 10 14:03:42.205577 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.205455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs\") pod \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\" (UID: \"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a\") "
Sep 10 14:03:42.205796 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.205759    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" (UID: "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:03:42.206862 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.206803    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:42.207904 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.207867    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m" (OuterVolumeSpecName: "kube-api-access-gv74m") pod "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" (UID: "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a"). InnerVolumeSpecName "kube-api-access-gv74m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:03:42.208116 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.208088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" (UID: "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:42.208116 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.208100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" (UID: "fe88808a-4d8e-44b8-b4c1-0e6d13982b7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:03:42.308101 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.308014    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv74m\" (UniqueName: \"kubernetes.io/projected/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-kube-api-access-gv74m\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:42.308101 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.308047    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:42.308101 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.308055    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:03:42.948097 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.948066    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"
Sep 10 14:03:42.948097 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.948074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7" event={"ID":"fe88808a-4d8e-44b8-b4c1-0e6d13982b7a","Type":"ContainerDied","Data":"983a8a179781f2e118ffdcbb88f48498edcb85303c0364e907e619aafc37d03f"}
Sep 10 14:03:42.948636 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.948161    2577 scope.go:117] "RemoveContainer" containerID="b19dd7c22bbe2462e7dc03d65dbdc8b11c8b807fe06e5e78433cb0521f75301a"
Sep 10 14:03:42.992529 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.992468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 14:03:42.995529 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:42.995476    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-a00003-76b46cf9b7-8rsl7"]
Sep 10 14:03:44.063543 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:44.063489    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" path="/var/lib/kubelet/pods/fe88808a-4d8e-44b8-b4c1-0e6d13982b7a/volumes"
Sep 10 14:03:44.561710 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:44.561678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:03:54.445483 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:03:54.445445    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:05:16.341547 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:05:16.341494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:05:45.148858 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:05:45.148817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:05:52.265613 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:05:52.265576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:05:58.910005 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:05:58.909912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:06:07.050725 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:06:07.050684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:06:24.738202 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:06:24.738162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:06:29.745717 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:06:29.745679    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:06:35.942859 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:06:35.942815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:08:06.408694 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:08:06.408656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:08:15.299807 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:08:15.299765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:08:36.197991 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:08:36.197960    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 14:08:36.204896 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:08:36.204859    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hjbdc_7567df29-f6d2-49d8-9bc8-b793630a47f1/kube-multus/2.log"
Sep 10 14:08:48.755094 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:08:48.755059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:09:00.164189 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164110    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164490    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164503    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" containerName="maas-api"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" containerName="maas-api"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a61137-9283-4646-a8c1-36176b99f799" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164546    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a61137-9283-4646-a8c1-36176b99f799" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164554    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f914eea1-c419-4c92-ba25-edf4a5505a93" containerName="maas-api"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164574    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f914eea1-c419-4c92-ba25-edf4a5505a93" containerName="maas-api"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da858bcf-76c2-4619-b149-7bd48f224a51" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164590    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da858bcf-76c2-4619-b149-7bd48f224a51" containerName="payload-pre-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164598    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="080f8b9b-c6bf-449d-848c-5719e9c059e3" containerName="main"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164603    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="080f8b9b-c6bf-449d-848c-5719e9c059e3" containerName="main"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164614    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" containerName="payload-processing"
Sep 10 14:09:00.164662 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164620    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" containerName="payload-processing"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164688    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="080f8b9b-c6bf-449d-848c-5719e9c059e3" containerName="main"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164695    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a61137-9283-4646-a8c1-36176b99f799" containerName="payload-pre-processing"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164700    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe88808a-4d8e-44b8-b4c1-0e6d13982b7a" containerName="maas-api"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164706    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da858bcf-76c2-4619-b149-7bd48f224a51" containerName="payload-pre-processing"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164711    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fcf1dab-ef1d-4baf-8b71-b031c4e3474d" containerName="payload-pre-processing"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164718    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f914eea1-c419-4c92-ba25-edf4a5505a93" containerName="maas-api"
Sep 10 14:09:00.165184 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.164724    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b4203-4c8f-4cfc-9f97-57df0ec57e6e" containerName="payload-processing"
Sep 10 14:09:00.167883 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.167863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.175624 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.175600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-878b46e6-kserve-self-signed-certs\""
Sep 10 14:09:00.189061 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.189028    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:09:00.222349 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.222551 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9x6\" (UniqueName: \"kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.222551 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222479    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.222551 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.222685 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.222685 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.222633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323276 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323276 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323558 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323558 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323558 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323737 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9x6\" (UniqueName: \"kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323862 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323927 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.323927 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.323909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.325602 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.325574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.325992 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.325972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.338933 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.338905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9x6\" (UniqueName: \"kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6\") pod \"e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") " pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.476911 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.476808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:00.777175 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:00.777147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:09:00.779750 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:09:00.779717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef27444_ff8c_47f0_ac5c_2c5f676f6ece.slice/crio-81acf3a98fc060dffb96a3c4889da8549ad51eed0dd3bcf9ee80c494e362652b WatchSource:0}: Error finding container 81acf3a98fc060dffb96a3c4889da8549ad51eed0dd3bcf9ee80c494e362652b: Status 404 returned error can't find the container with id 81acf3a98fc060dffb96a3c4889da8549ad51eed0dd3bcf9ee80c494e362652b
Sep 10 14:09:01.458588 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:01.458552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" event={"ID":"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece","Type":"ContainerStarted","Data":"9ed3bc690517923ee7462cbb51bc97898a8a971b316aa69332e53d984a286711"}
Sep 10 14:09:01.458588 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:01.458593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" event={"ID":"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece","Type":"ContainerStarted","Data":"81acf3a98fc060dffb96a3c4889da8549ad51eed0dd3bcf9ee80c494e362652b"}
Sep 10 14:09:01.512735 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:01.512621    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" podStartSLOduration=1.512604279 podStartE2EDuration="1.512604279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:09:01.512287433 +0000 UTC m=+2425.967162596" watchObservedRunningTime="2026-09-10 14:09:01.512604279 +0000 UTC m=+2425.967479440"
Sep 10 14:09:03.256866 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:03.256823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:09:10.477770 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:10.477735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:10.478271 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:10.478065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:10.490279 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:10.490257    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:10.516556 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:10.516530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:09:49.537703 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:49.537665    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:09:51.862249 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:51.862210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:09:59.653756 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:09:59.653718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:04.558218 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.558162    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" podUID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" containerName="main" containerID="cri-o://9ed3bc690517923ee7462cbb51bc97898a8a971b316aa69332e53d984a286711" gracePeriod=45
Sep 10 14:10:04.766222 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.766185    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" containerID="9ed3bc690517923ee7462cbb51bc97898a8a971b316aa69332e53d984a286711" exitCode=0
Sep 10 14:10:04.766389 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.766262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" event={"ID":"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece","Type":"ContainerDied","Data":"9ed3bc690517923ee7462cbb51bc97898a8a971b316aa69332e53d984a286711"}
Sep 10 14:10:04.809349 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.809292    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:10:04.934705 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.934909 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934713    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.934909 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf9x6\" (UniqueName: \"kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.934909 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.934909 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.934909 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs\") pod \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\" (UID: \"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece\") "
Sep 10 14:10:04.935160 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:10:04.935160 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.934982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:10:04.935160 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.935034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home" (OuterVolumeSpecName: "home") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:10:04.935313 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.935229    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-home\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:04.935313 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.935246    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tmp-dir\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:04.935313 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.935256    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-model-cache\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:04.936961 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.936932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6" (OuterVolumeSpecName: "kube-api-access-sf9x6") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "kube-api-access-sf9x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:10:04.936961 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.936955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:10:04.937131 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:04.937011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm" (OuterVolumeSpecName: "dshm") pod "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" (UID: "5ef27444-ff8c-47f0-ac5c-2c5f676f6ece"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:10:05.036333 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.036295    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9x6\" (UniqueName: \"kubernetes.io/projected/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-kube-api-access-sf9x6\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:05.036333 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.036323    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-dshm\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:05.036333 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.036334    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece-tls-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:05.263034 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.262996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:05.772564 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.772531    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"
Sep 10 14:10:05.772564 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.772544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw" event={"ID":"5ef27444-ff8c-47f0-ac5c-2c5f676f6ece","Type":"ContainerDied","Data":"81acf3a98fc060dffb96a3c4889da8549ad51eed0dd3bcf9ee80c494e362652b"}
Sep 10 14:10:05.773070 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.772605    2577 scope.go:117] "RemoveContainer" containerID="9ed3bc690517923ee7462cbb51bc97898a8a971b316aa69332e53d984a286711"
Sep 10 14:10:05.794791 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.794767    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:10:05.798868 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:05.798844    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-878b46e6-kserve-6ff9b84c75-tzmdw"]
Sep 10 14:10:06.063925 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:06.063841    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" path="/var/lib/kubelet/pods/5ef27444-ff8c-47f0-ac5c-2c5f676f6ece/volumes"
Sep 10 14:10:12.453260 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:12.453217    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:13.054062 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054027    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"]
Sep 10 14:10:13.054405 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054390    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" containerName="main"
Sep 10 14:10:13.054405 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054403    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" containerName="main"
Sep 10 14:10:13.054592 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054420    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:10:13.054592 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054426    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:10:13.054592 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054481    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef27444-ff8c-47f0-ac5c-2c5f676f6ece" containerName="main"
Sep 10 14:10:13.054592 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.054490    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2055234-c5c4-472b-95ac-5147dac379bc" containerName="istio-proxy"
Sep 10 14:10:13.058760 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.058732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.068021 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.067993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"]
Sep 10 14:10:13.213910 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.213871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-metrics-certs\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.214124 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.213955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-maas-api-tls\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.214124 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.214083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16707d22-dcbf-4c60-b925-a49e97827c4d-openshift-service-ca\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.214124 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.214118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds2k\" (UniqueName: \"kubernetes.io/projected/16707d22-dcbf-4c60-b925-a49e97827c4d-kube-api-access-9ds2k\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.315422 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.315315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-maas-api-tls\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.315625 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.315431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16707d22-dcbf-4c60-b925-a49e97827c4d-openshift-service-ca\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.315625 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.315460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds2k\" (UniqueName: \"kubernetes.io/projected/16707d22-dcbf-4c60-b925-a49e97827c4d-kube-api-access-9ds2k\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.315625 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.315487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-metrics-certs\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.316182 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.316155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16707d22-dcbf-4c60-b925-a49e97827c4d-openshift-service-ca\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.317806 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.317783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-metrics-certs\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.317904 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.317877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16707d22-dcbf-4c60-b925-a49e97827c4d-maas-api-tls\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.323752 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.323727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds2k\" (UniqueName: \"kubernetes.io/projected/16707d22-dcbf-4c60-b925-a49e97827c4d-kube-api-access-9ds2k\") pod \"maas-api-67ff8f498c-z6lqq\" (UID: \"16707d22-dcbf-4c60-b925-a49e97827c4d\") " pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.370441 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.370406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:13.645266 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.645240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"]
Sep 10 14:10:13.647137 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:10:13.647104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16707d22_dcbf_4c60_b925_a49e97827c4d.slice/crio-a523055b4fca1a90304f1c6139036dd0dedee0df7f1658b144a6b879afe8e591 WatchSource:0}: Error finding container a523055b4fca1a90304f1c6139036dd0dedee0df7f1658b144a6b879afe8e591: Status 404 returned error can't find the container with id a523055b4fca1a90304f1c6139036dd0dedee0df7f1658b144a6b879afe8e591
Sep 10 14:10:13.648651 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.648630    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:10:13.812107 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:13.812067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq" event={"ID":"16707d22-dcbf-4c60-b925-a49e97827c4d","Type":"ContainerStarted","Data":"a523055b4fca1a90304f1c6139036dd0dedee0df7f1658b144a6b879afe8e591"}
Sep 10 14:10:14.819287 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:14.819251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq" event={"ID":"16707d22-dcbf-4c60-b925-a49e97827c4d","Type":"ContainerStarted","Data":"1734358a59301e7b9fff4ee8e5650e4569e54e5180666580e4612a3f4f2f30a8"}
Sep 10 14:10:14.819725 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:14.819313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:14.847034 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:14.846983    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq" podStartSLOduration=1.433538163 podStartE2EDuration="1.846965299s" totalImagesPullingTime="413.427136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:10:14.844463694 +0000 UTC m=+2499.299338857" watchObservedRunningTime="2026-09-10 14:10:14.846965299 +0000 UTC m=+2499.301840494"
Sep 10 14:10:20.650488 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:20.650451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:20.828736 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:20.828708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67ff8f498c-z6lqq"
Sep 10 14:10:20.887346 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:20.887312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 14:10:20.887615 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:20.887581    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerName="maas-api" containerID="cri-o://e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186" gracePeriod=30
Sep 10 14:10:22.127957 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.127932    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 14:10:22.199670 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.199637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca\") pod \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") "
Sep 10 14:10:22.199861 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.199701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs\") pod \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") "
Sep 10 14:10:22.199861 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.199768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpj7d\" (UniqueName: \"kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d\") pod \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") "
Sep 10 14:10:22.199861 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.199789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls\") pod \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\" (UID: \"8fb2670e-2045-4af3-b1db-00c4fa7faf8b\") "
Sep 10 14:10:22.200088 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.200063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fb2670e-2045-4af3-b1db-00c4fa7faf8b" (UID: "8fb2670e-2045-4af3-b1db-00c4fa7faf8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:10:22.201922 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.201890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fb2670e-2045-4af3-b1db-00c4fa7faf8b" (UID: "8fb2670e-2045-4af3-b1db-00c4fa7faf8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:10:22.201922 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.201900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d" (OuterVolumeSpecName: "kube-api-access-qpj7d") pod "8fb2670e-2045-4af3-b1db-00c4fa7faf8b" (UID: "8fb2670e-2045-4af3-b1db-00c4fa7faf8b"). InnerVolumeSpecName "kube-api-access-qpj7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:10:22.202073 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.201924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8fb2670e-2045-4af3-b1db-00c4fa7faf8b" (UID: "8fb2670e-2045-4af3-b1db-00c4fa7faf8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:10:22.301223 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.301126    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-openshift-service-ca\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:22.301223 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.301161    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-metrics-certs\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:22.301223 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.301173    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpj7d\" (UniqueName: \"kubernetes.io/projected/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-kube-api-access-qpj7d\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:22.301223 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.301185    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb2670e-2045-4af3-b1db-00c4fa7faf8b-maas-api-tls\") on node \"ip-10-0-133-48.ec2.internal\" DevicePath \"\""
Sep 10 14:10:22.859250 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.859216    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerID="e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186" exitCode=0
Sep 10 14:10:22.859428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.859262    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"
Sep 10 14:10:22.859428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.859301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" event={"ID":"8fb2670e-2045-4af3-b1db-00c4fa7faf8b","Type":"ContainerDied","Data":"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"}
Sep 10 14:10:22.859428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.859352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" event={"ID":"8fb2670e-2045-4af3-b1db-00c4fa7faf8b","Type":"ContainerDied","Data":"fc712267ee6908ab17b0f663f7326d647f5e40cb658bbbf1051e748f24fe01d6"}
Sep 10 14:10:22.859428 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.859375    2577 scope.go:117] "RemoveContainer" containerID="e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"
Sep 10 14:10:22.868523 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.868495    2577 scope.go:117] "RemoveContainer" containerID="e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"
Sep 10 14:10:22.868783 ip-10-0-133-48 kubenswrapper[2577]: E0910 14:10:22.868761    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186\": container with ID starting with e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186 not found: ID does not exist" containerID="e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"
Sep 10 14:10:22.868860 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.868790    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186"} err="failed to get container status \"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186\": rpc error: code = NotFound desc = could not find container \"e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186\": container with ID starting with e14a30b6c66ac03abdb36d6bae521fe0f9537c47751bf8330447a6a7bb610186 not found: ID does not exist"
Sep 10 14:10:22.882106 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.882080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 14:10:22.887480 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:22.887460    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk"]
Sep 10 14:10:24.064717 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:24.064677    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" path="/var/lib/kubelet/pods/8fb2670e-2045-4af3-b1db-00c4fa7faf8b/volumes"
Sep 10 14:10:25.086016 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:25.085968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-cb9d756dc-9lfqk" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.33:8443/health\": context deadline exceeded"
Sep 10 14:10:27.335854 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:27.335824    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:33.353637 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:33.353548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:40.244081 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:40.244044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-2vv9k"]
Sep 10 14:10:48.107695 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:48.107662    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-722tv_ca23f6d2-928f-412e-8e52-b5f9be9a3f85/manager/0.log"
Sep 10 14:10:48.220520 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:48.220473    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wr98b_f77aec85-e054-4d1e-a67a-a946072310ec/manager/0.log"
Sep 10 14:10:48.989845 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:48.989810    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-kzx4f_9a6382dc-365b-46b3-9a9b-1a8a0a342154/copy-manifests/0.log"
Sep 10 14:10:48.997004 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:48.996986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-kzx4f_9a6382dc-365b-46b3-9a9b-1a8a0a342154/manager/0.log"
Sep 10 14:10:49.469082 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:49.469037    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67ff8f498c-z6lqq_16707d22-dcbf-4c60-b925-a49e97827c4d/maas-api/0.log"
Sep 10 14:10:49.834182 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:49.834102    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-9zwrv_a4e831e9-f4c9-45fc-98dc-51c3fd18e10c/postgres/0.log"
Sep 10 14:10:50.715784 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:50.715751    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-n7mn9_f086ce38-0404-4b44-916e-48515aa6c593/manager/0.log"
Sep 10 14:10:50.832090 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:50.832060    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2blzk_acc559a0-90cf-4547-840c-096c9d24dafb/manager/0.log"
Sep 10 14:10:50.943844 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:50.943815    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-t7glb_7f68e2ce-7a1d-47df-8e65-db2edeef86e8/kuadrant-console-plugin/0.log"
Sep 10 14:10:51.190569 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:51.190534    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2vv9k_4c17b355-dbf5-43b2-8c65-1388a68da666/limitador/0.log"
Sep 10 14:10:51.321182 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:51.321152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lrg89_d116ee42-0148-4339-b471-a6b63d88d58b/manager/0.log"
Sep 10 14:10:52.044310 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:52.044281    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-jr7wq_853e7098-e0ef-4923-a8a9-0cbd513d423d/kube-auth-proxy/0.log"
Sep 10 14:10:52.165916 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:52.165883    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/1.log"
Sep 10 14:10:53.203354 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:53.203323    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9jlr8_cfc95104-471f-41a2-afea-ab36f7a614d3/main/0.log"
Sep 10 14:10:53.425231 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:53.425192    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cfvpx_0a08ade6-203f-42ae-9224-ef0f5495e5ab/main/0.log"
Sep 10 14:10:53.537037 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:53.536946    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6pplx_0e8e1d2d-9318-4e9b-9638-800859c6eaf6/main/0.log"
Sep 10 14:10:59.040365 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.040333    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5vxtc/must-gather-8l2vt"]
Sep 10 14:10:59.040792 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.040698    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerName="maas-api"
Sep 10 14:10:59.040792 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.040709    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerName="maas-api"
Sep 10 14:10:59.040792 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.040770    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb2670e-2045-4af3-b1db-00c4fa7faf8b" containerName="maas-api"
Sep 10 14:10:59.044004 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.043982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.046159 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.046138    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"kube-root-ca.crt\""
Sep 10 14:10:59.046278 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.046233    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5vxtc\"/\"default-dockercfg-mpghg\""
Sep 10 14:10:59.046738 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.046724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5vxtc\"/\"openshift-service-ca.crt\""
Sep 10 14:10:59.053939 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.053912    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/must-gather-8l2vt"]
Sep 10 14:10:59.130220 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.130186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwvw\" (UniqueName: \"kubernetes.io/projected/50529b96-78fd-453f-8cd3-3516ff1aa645-kube-api-access-rdwvw\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.130419 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.130312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50529b96-78fd-453f-8cd3-3516ff1aa645-must-gather-output\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.230786 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.230748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50529b96-78fd-453f-8cd3-3516ff1aa645-must-gather-output\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.230974 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.230810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwvw\" (UniqueName: \"kubernetes.io/projected/50529b96-78fd-453f-8cd3-3516ff1aa645-kube-api-access-rdwvw\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.231106 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.231080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50529b96-78fd-453f-8cd3-3516ff1aa645-must-gather-output\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.245246 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.245219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwvw\" (UniqueName: \"kubernetes.io/projected/50529b96-78fd-453f-8cd3-3516ff1aa645-kube-api-access-rdwvw\") pod \"must-gather-8l2vt\" (UID: \"50529b96-78fd-453f-8cd3-3516ff1aa645\") " pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.354500 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.354406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/must-gather-8l2vt"
Sep 10 14:10:59.630116 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:10:59.630083    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/must-gather-8l2vt"]
Sep 10 14:10:59.632704 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:10:59.632678    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50529b96_78fd_453f_8cd3_3516ff1aa645.slice/crio-f45db5f2e596833a9a36e6f3e5a635770f8fbbb48afc086ada748a31034d5b20 WatchSource:0}: Error finding container f45db5f2e596833a9a36e6f3e5a635770f8fbbb48afc086ada748a31034d5b20: Status 404 returned error can't find the container with id f45db5f2e596833a9a36e6f3e5a635770f8fbbb48afc086ada748a31034d5b20
Sep 10 14:11:00.037342 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:00.037252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/must-gather-8l2vt" event={"ID":"50529b96-78fd-453f-8cd3-3516ff1aa645","Type":"ContainerStarted","Data":"f45db5f2e596833a9a36e6f3e5a635770f8fbbb48afc086ada748a31034d5b20"}
Sep 10 14:11:03.056529 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:03.056467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/must-gather-8l2vt" event={"ID":"50529b96-78fd-453f-8cd3-3516ff1aa645","Type":"ContainerStarted","Data":"73f609e9ac6ee4b78ca90930495f2e52040c92276fcacac8c16ae9175e78fa62"}
Sep 10 14:11:03.057047 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:03.056539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/must-gather-8l2vt" event={"ID":"50529b96-78fd-453f-8cd3-3516ff1aa645","Type":"ContainerStarted","Data":"99c2d7a02c2748d58d2b6d67c01e51c4bda8af237d65b3f993695f3777c25892"}
Sep 10 14:11:03.080015 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:03.079947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5vxtc/must-gather-8l2vt" podStartSLOduration=1.39115256 podStartE2EDuration="4.07992361s" totalImagesPullingTime="2.68877105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:11:03.079070947 +0000 UTC m=+2547.533946111" watchObservedRunningTime="2026-09-10 14:11:03.07992361 +0000 UTC m=+2547.534798775"
Sep 10 14:11:04.220495 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:04.220462    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9slnm_13a758ab-ae02-4484-b6f1-0773f7e2a50a/global-pull-secret-syncer/0.log"
Sep 10 14:11:04.498538 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:04.498445    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sdzbv_3ac85a2f-e430-4f7a-8db3-8b2f6b345143/konnectivity-agent/0.log"
Sep 10 14:11:04.527734 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:04.527696    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-48.ec2.internal_13f9881a5f465bf70dd944de0203fca5/haproxy/0.log"
Sep 10 14:11:08.948600 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:08.948559    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-n7mn9_f086ce38-0404-4b44-916e-48515aa6c593/manager/0.log"
Sep 10 14:11:08.987725 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:08.987660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2blzk_acc559a0-90cf-4547-840c-096c9d24dafb/manager/0.log"
Sep 10 14:11:09.023629 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:09.023590    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-t7glb_7f68e2ce-7a1d-47df-8e65-db2edeef86e8/kuadrant-console-plugin/0.log"
Sep 10 14:11:09.106029 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:09.105999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-2vv9k_4c17b355-dbf5-43b2-8c65-1388a68da666/limitador/0.log"
Sep 10 14:11:09.219656 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:09.219483    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lrg89_d116ee42-0148-4339-b471-a6b63d88d58b/manager/0.log"
Sep 10 14:11:11.132173 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.132137    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-f7mw5_6dd35db0-2c03-4be2-99ae-8592b784cde1/kube-state-metrics/0.log"
Sep 10 14:11:11.185021 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.184938    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-f7mw5_6dd35db0-2c03-4be2-99ae-8592b784cde1/kube-rbac-proxy-main/0.log"
Sep 10 14:11:11.275259 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.275204    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-f7mw5_6dd35db0-2c03-4be2-99ae-8592b784cde1/kube-rbac-proxy-self/0.log"
Sep 10 14:11:11.444740 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.444664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5gb7_e2644f21-2f03-459e-b7c0-c766136b6162/node-exporter/0.log"
Sep 10 14:11:11.495407 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.495347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5gb7_e2644f21-2f03-459e-b7c0-c766136b6162/kube-rbac-proxy/0.log"
Sep 10 14:11:11.564459 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:11.564427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l5gb7_e2644f21-2f03-459e-b7c0-c766136b6162/init-textfile/0.log"
Sep 10 14:11:12.101203 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.101168    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/prometheus/0.log"
Sep 10 14:11:12.146916 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.146834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/config-reloader/0.log"
Sep 10 14:11:12.194910 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.194873    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/thanos-sidecar/0.log"
Sep 10 14:11:12.240522 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.240478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/kube-rbac-proxy-web/0.log"
Sep 10 14:11:12.294106 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.294073    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/kube-rbac-proxy/0.log"
Sep 10 14:11:12.346802 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.346763    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/kube-rbac-proxy-thanos/0.log"
Sep 10 14:11:12.394690 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.394661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ed9b0e77-0378-404d-ac0a-9eb6c5ff92af/init-config-reloader/0.log"
Sep 10 14:11:12.617699 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:12.617669    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-vg7rr_2960223e-bcff-4d5c-8f04-2abde821af3e/prometheus-operator-admission-webhook/0.log"
Sep 10 14:11:14.162685 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.162641    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"]
Sep 10 14:11:14.168066 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.168035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.176417 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.176391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"]
Sep 10 14:11:14.294340 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.294287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvsq\" (UniqueName: \"kubernetes.io/projected/ea24e867-5610-4d00-8b61-21c683e12de3-kube-api-access-mxvsq\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.294787 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.294761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-lib-modules\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.295002 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.294965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-podres\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.295200 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.295164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-sys\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.295383 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.295352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-proc\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396418 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-sys\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396418 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-proc\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvsq\" (UniqueName: \"kubernetes.io/projected/ea24e867-5610-4d00-8b61-21c683e12de3-kube-api-access-mxvsq\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-sys\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-lib-modules\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-podres\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-proc\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-podres\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.396764 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.396745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea24e867-5610-4d00-8b61-21c683e12de3-lib-modules\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.414172 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.414087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvsq\" (UniqueName: \"kubernetes.io/projected/ea24e867-5610-4d00-8b61-21c683e12de3-kube-api-access-mxvsq\") pod \"perf-node-gather-daemonset-8nxcj\" (UID: \"ea24e867-5610-4d00-8b61-21c683e12de3\") " pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.480927 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.480890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:14.776378 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:14.776337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"]
Sep 10 14:11:14.780091 ip-10-0-133-48 kubenswrapper[2577]: W0910 14:11:14.780053    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea24e867_5610_4d00_8b61_21c683e12de3.slice/crio-b14a6f207897093098b61965e39d630eb2be0217e72b0981183c8e561bd3331c WatchSource:0}: Error finding container b14a6f207897093098b61965e39d630eb2be0217e72b0981183c8e561bd3331c: Status 404 returned error can't find the container with id b14a6f207897093098b61965e39d630eb2be0217e72b0981183c8e561bd3331c
Sep 10 14:11:15.140816 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:15.140770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj" event={"ID":"ea24e867-5610-4d00-8b61-21c683e12de3","Type":"ContainerStarted","Data":"862aa60cbbd0f47e355040940b1b03352f65dd0dc51bcda9268848457d11c45f"}
Sep 10 14:11:15.141039 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:15.140856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:15.141039 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:15.140887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj" event={"ID":"ea24e867-5610-4d00-8b61-21c683e12de3","Type":"ContainerStarted","Data":"b14a6f207897093098b61965e39d630eb2be0217e72b0981183c8e561bd3331c"}
Sep 10 14:11:15.165141 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:15.165072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj" podStartSLOduration=1.165047426 podStartE2EDuration="1.165047426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:11:15.162611995 +0000 UTC m=+2559.617487158" watchObservedRunningTime="2026-09-10 14:11:15.165047426 +0000 UTC m=+2559.619922590"
Sep 10 14:11:17.579061 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:17.579029    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-skb2d_b9f15e82-4927-42fc-afa5-baa414a87a5b/dns/0.log"
Sep 10 14:11:17.607747 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:17.607710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-skb2d_b9f15e82-4927-42fc-afa5-baa414a87a5b/kube-rbac-proxy/0.log"
Sep 10 14:11:17.707671 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:17.707641    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wl48r_317a825d-39ad-4035-bffe-ac32f2be5000/dns-node-resolver/0.log"
Sep 10 14:11:18.347423 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:18.347396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f4849c767-t627c_8f80ae62-51b3-4e45-8b28-a8c7b74da82a/registry/0.log"
Sep 10 14:11:18.411735 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:18.411707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nkwgj_39c2f8f3-bd46-48e1-b1b5-b7a07d99bf9e/node-ca/0.log"
Sep 10 14:11:20.044005 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:20.043975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-jr7wq_853e7098-e0ef-4923-a8a9-0cbd513d423d/kube-auth-proxy/0.log"
Sep 10 14:11:20.077755 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:20.077720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/0.log"
Sep 10 14:11:20.088665 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:20.088633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/1.log"
Sep 10 14:11:21.159313 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:21.159284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5vxtc/perf-node-gather-daemonset-8nxcj"
Sep 10 14:11:21.168503 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:21.168475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b2vwx_614bc448-cf0b-40b7-8bfe-28664f494c7b/serve-healthcheck-canary/0.log"
Sep 10 14:11:21.897250 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:21.897223    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84ts4_f640fe27-8902-4eb0-982f-11af5032e789/kube-rbac-proxy/0.log"
Sep 10 14:11:21.924798 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:21.924762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84ts4_f640fe27-8902-4eb0-982f-11af5032e789/exporter/0.log"
Sep 10 14:11:21.953841 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:21.953813    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84ts4_f640fe27-8902-4eb0-982f-11af5032e789/extractor/0.log"
Sep 10 14:11:24.650269 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:24.650233    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-722tv_ca23f6d2-928f-412e-8e52-b5f9be9a3f85/manager/0.log"
Sep 10 14:11:24.878285 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:24.878247    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wr98b_f77aec85-e054-4d1e-a67a-a946072310ec/manager/0.log"
Sep 10 14:11:25.023287 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:25.023262    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-jr7wq_853e7098-e0ef-4923-a8a9-0cbd513d423d/kube-auth-proxy/0.log"
Sep 10 14:11:25.065039 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:25.065007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/0.log"
Sep 10 14:11:25.073082 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:25.073058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/1.log"
Sep 10 14:11:25.456251 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:25.456222    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-kzx4f_9a6382dc-365b-46b3-9a9b-1a8a0a342154/manager/0.log"
Sep 10 14:11:25.531253 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:25.531225    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-579d56558b-kzx4f_9a6382dc-365b-46b3-9a9b-1a8a0a342154/copy-manifests/0.log"
Sep 10 14:11:27.297275 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:27.297244    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69ddc5c8c5-jr7wq_853e7098-e0ef-4923-a8a9-0cbd513d423d/kube-auth-proxy/0.log"
Sep 10 14:11:27.325014 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:27.324984    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/0.log"
Sep 10 14:11:27.335300 ip-10-0-133-48 kubenswrapper[2577]: I0910 14:11:27.335263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v29wf_1dca92b2-0418-47d9-a342-0f2e4d21c55d/istio-proxy/1.log"